Metering user behaviour and engagement with user interface in terminal devices

• US 10,084,869 B2
• Filed: 10/04/2013
• Issued: 09/25/2018
• Est. Priority Date: 10/04/2013
• Status: Active Grant

• Current Assignee: Verto Analytics Oy (Verto Analytics, Inc.)
• Original Assignee: Verto Analytics Oy (Verto Analytics, Inc.)
• Inventors: Verkasalo, Hannu
• Primary Examiner(s): Kawsar, Abdullah Al
• Assistant Examiner(s): Luu, David V
• Application Number:
  US14/046,502
• Publication Number:
  US 20150100887A1
• Time in Patent Office: 1,817 Days
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06Q 10/00 Administration; Management
• CPC Class Codes: H04L 41/0803 Configuration setting
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

Electronic terminal device including a processing entity for processing instructions and other data, functionally connected user interface entity, a display and user input elements, optionally a keypad or a touch-sensitive arrangement in connection with the display, a memory entity for storing the instructions and data, and a communication interface for transferring data with external entities, configured, in accordance with the instructions, to store data including a number of predetermined fingerprints, each characterizing display usage such as visual objects associated with software entities and potentially shown to the user via the display during execution of the software entity, and log data including behavioral data indicative of user engagement with the UI of the device, the logging incorporating detecting display usage, and storing indication of such detection in the memory, wherein the device is configured to determine the status of the display, and execute the logging according to the determination.

Claims

1. An electronic terminal device, comprising:

a processor that processes processor-executable instructions and other data, at least functionally connected to each of

a user interface (UI) apparatus that interfaces the electronic terminal device with a user, the UI apparatus comprising a display and a user input device,

a memory that stores the instructions and other data, and

a communication interface that transfers data between the processor and external entities,

the processor being configured, in accordance with the processor-executable instructions, to;

store, in the memory, data comprising a plurality of digital content fingerprints that each correspond to a usage of display content, said display content being displayed on the display based on instructions of one or more software entities that operate on the electronic terminal device; and

generate and record log data, comprising behavioral data indicative of engagement by a user with the UI apparatus of the electronic terminal device, said log data generated by carrying out detection and logging of usage, by a user, of graphical objects rendered on the display in accordance with the instructions of the one or more software entities that operate on the electronic terminal device, and searching for and detecting one or more matches between the stored digital content fingerprints and the detected usage of the display content by the user generated and stored as the log data, and

storing indication of said detection of said one or more matches in said memory,

wherein the processor is further configured i) to determine a status of the display, and ii) to carry out the generating and logging of the log data according to the determined status of the display,

wherein the determined status of the display indicates, at least, whether or not the display is in active use by the user,

wherein, upon a determination from the status that the display is in active use by the user, the generating and recording of the log data is performed, and

wherein, upon a determination from the status that the display is not in use, the processor continues to generate and record said log data, and further i) generates metadata and ii) includes said metadata with said log data,

said included metadata being an indicator that said log data is data collected and logged when the display is not in use,

and wherein the processor is further configured to interact with and receive data from at least one external device, said data received being indicative of user engagement with the UI of the external device, and to determine an indication of a level of user attention spread across the electronic terminal device and said at least one external device, where one of the electronic terminal device and said at least one external device is determined as capturing a highest level of user attention.

2. The electronic terminal device of claim 1, wherein said digital content fingerprints comprise information representative of one or more visual objects associated with one or more software entities, said visual objects shown to the user via the display, and said detection of usage of the display incorporates detecting a presence of any of said objects rendered on the display.

3. The electronic terminal device of claim 2, wherein the processor is further configured to detect said presence by utilizing screen capture of display data and performing an analysis of the obtained display data by performing a comparison of the obtained display data with the stored digital content fingerprints.

4. The electronic terminal device of claim 3, wherein the analysis of the obtained display data comprises execution of a predetermined pattern recognition algorithm.

5. The electronic terminal device of claim 1, wherein the generating and recording the log data comprises at least one control action selected from the group consisting of:

including an indication of the determined display status in the logged data, adapting said logging, trigger logging, trigger logging of behavioral data, trigger logging of predetermined data, cease logging, cease logging of behavioral data, cease logging of predetermined data, change logging parameter, and trigger said detecting.

6. The electronic terminal device of claim 1, wherein the processor is further configured to detect one or more matches between the stored digital content fingerprints and one or more software entities shown to the user via the display in accordance with instructions of one or more software entities that operate on the plurality of electronic terminal devices, said detecting taking place exclusively when the display has been determined to be in use.

7. The electronic terminal device of claim 1, wherein the processor is further configured to determine the status of the display based on results of carrying out an inspection of at least one indication associated with the display selected from the group consisting of:

display power or current consumption, display brightness, display lock status, a user input interface or user interface lock status, a user input, a display status-indicating application programming interface (API), and a display status variable.

8. The electronic terminal device of claim 1, wherein the processor is further configured to generate and record log data regarding at least one element selected from the group consisting of:

user online action, HTTP activity, a sensor, acceleration, accelerometer, illumination or light intensity, application usage, application launch, application termination, user id, process thread, identity of application or service used, a feature used in application or service, device-internal data traffic, network traffic, a source or destination entity of internal and/or network data traffic, a voice call, an application thread counter, a process counter, a message, an e-mail message, a short message, a multimedia message, an instant message, a data counter, a location, battery status, and network signal strength.

9. The electronic terminal device of claim 1, wherein the indication of said detection comprises metadata associated with the stored digital content fingerprints detected as matching with the detected usage of the display content by the user.

10. The electronic terminal device of claim 1, wherein the indication of said detection comprises at least one element selected from the group consisting of:

type of display view, active window of multiple simultaneously visible windows, identity of visible windows, identity of the existing windows, identity of application or service in the active window, duration and/or time of use of application or service, transaction, feature or view accessed in the displayed application or service, and user activity regarding application, service or display.

11. The electronic terminal device of claim 1, wherein the processor is further configured to communicate the indication of said detection of said one or more matches to a predetermined external entity for storage, analysis, inspection or distribution.

12. The electronic terminal device of claim 1, wherein the processor is further configured to adapt the generating and recording of the log data based on at least one element selected from the group consisting of:

battery status, remaining battery, remaining operating time, and network status.

13. The electronic terminal device of claim 1, wherein the processor is further configured to adapt the generating and recording of the log data through execution of at least one action selected from the group consisting of:

triggering collection of predetermined data, interrupting collection of predetermined data, triggering predetermined data processing or analysis procedure, interrupting predetermined data processing or analysis procedure, triggering data transfer with an external entity, and triggering more detailed or coarser detection of objects on the display.

14. The electronic terminal device of claim 1, wherein the processor is configured to detect a transaction based on at least one object rendered on the display data and a match of said transaction with one or more of the plurality of digital content fingerprints,

whereupon the processor is further configured to log data regarding the transaction.

15. The electronic terminal device of claim 1, wherein the processor is further configured to monitor whether said generating and recording of log data is functional according to predetermined criterion, and upon a determination that said generating and recording of log data is not functional, the processor activates, reactivates and/or initializes said generating and recording of log data.

16. The electronic terminal device of claim 1, wherein the processor is configured to periodically execute a plurality of predetermined tasks to prevent an operating system of the electronic terminal device from shutting down said generating and recording of log data due to passivity or lack of user interaction.