Method, apparatus, and system for modeling passive and active user interactions with a computer system

• US 9,152,221 B2
• Filed: 09/28/2012
• Issued: 10/06/2015
• Est. Priority Date: 05/17/2012
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Denker, Grit, Senanayake, Rukman
• Primary Examiner(s): To, Jennifer
• Assistant Examiner(s): REPSHER III, JOHN T
• Application Number:
  US13/631,292
• Publication Number:
  US 20130311924A1
• Time in Patent Office: 1,103 Days
• Field of Search: 715/751, 715/831, 715/771
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2425 Iterative querying; Query formulation based on the results of a preceding query
• CPC Class Codes: G06F 16/332 Query formulation
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3349 Reuse of stored results of previous queries
• CPC Class Codes: G06F 16/90328 using search space presentation or visualization, e.g. category or range presentation and selection
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence

Abstract

A method, apparatus, and system for modeling user interactions with a computer system associates semantic descriptions of passive and active user interactions, which are meaningful at a user level, with application events and user interaction data as a user interacts with one or multiple software applications with a computing device, and uses those associations to build and maintain a user-specific contextual model. In some embodiments, the contextual models of multiple users are leveraged to form one or more collective contextual user models. Such models are useful in many different applications.

Claims

1. A method for modeling user activity with regard to a plurality of different software applications on a computing device, the method comprising, with the computing device:

generating semantic descriptions for each of a plurality of user interactions involving the plurality of different software applications over time, wherein (i) at least a first one of the semantic descriptions relates to gaze-tracking data comprising one or more real-time sensor inputs indicative of a user'"'"'s gaze in relation to a display of the computing device during a user interaction involving one of the plurality of different software applications; and

(ii) at least a second one of the semantic descriptions relates to application event data generated as a result of another interaction involving another one of the plurality of different software applications;

algorithmically drawing an inference about a current user activity with the computing device based on at least (i) the first one of the semantic descriptions relating to the gaze-tracking data obtained during the user interaction involving the one of the plurality of different software applications and (ii) the second one of the semantic descriptions relating to the application event data generated as a result of the other interaction involving the other one of the plurality of different software applications and (iii) a recency or frequency of user attention to a user interface element;

developing a model of user activity in relation to the computing device based on associations of the semantic descriptions with the gaze-tracking data over time, initially storing the associations in a short-term memory portion of the model, and moving at least a portion of the associations to a long-term memory portion of the model after a period of time; and

making the inference available for use by a computer-executable module of the computing device.

2. The method of claim 1, wherein the user interface element comprises one or more of text, graphic, video, interactive content, and an interactive control.

3. The method of claim 1, wherein at least one of the semantic descriptions comprises functional information about the user interface element.

4. The method of claim 1, wherein at least one of the semantic descriptions comprises information about content of the user interface element.

5. The method of claim 1, comprising identifying, from the plurality of different software applications, a software application associated with the user interface element.

6. The method of claim 5, comprising obtaining at least one of the semantic descriptions from the software application.

7. The method of claim 1, comprising storing the associations of the semantic descriptions with the gaze-tracking data over time and developing the model of user activity in relation to the computing device based on the stored associations.

8. The method of claim 7, wherein the stored associations relate to content viewed at the computing device.

9. The method of claim 7, wherein the semantic descriptions relate to user interface elements displayed by the plurality of different software applications, and the method comprises developing a model of user activity with respect to the plurality of different software applications based on the stored associations.

10. The method of claim 1, wherein the gaze-tracking data comprises data relating to the duration of user attention to the user interface element and the method comprises generating an inference based on the data relating to the duration of user attention to the user interface element.

11. The method of claim 10, wherein the inference based on the data relating to the duration of user attention to the user interface element relates to user cognitive processing of content of the user interface element.

12. The method of claim 1, comprising determining the period of time based on the recency of user attention to the user interface element, the time of day, demographic information, location, or context.

13. The method of claim 1, comprising receiving, from one or more of the plurality of different software applications, data relating to a user interaction with the user interface element and associating the data relating to the user interaction with the user interface element with the gaze-tracking data.

14. The method of claim 1, comprising associating a cognitive processing parameter relating to the gaze-tracking data, wherein the cognitive processing parameter indicates a degree of user cognitive processing of content of the user interface element.

15. The method of claim 1, comprising receiving the application event data from the other one of the plurality of different software applications, obtaining a semantic description of the application event, the semantic description of the application event comprising information about the application event, and using the association of the semantic description of the application event with the application event data to model user activity at the computing device.

16. The method of claim 1, comprising receiving hand position data comprising one or more real-time sensor inputs indicative of the proximity of at least a portion of a user'"'"'s hand in relation to a component of the computing device;

identifying an on-screen location corresponding to the hand position data;

obtaining a semantic description of a user interface element located at the on-screen location, the semantic description of the user interface element located at the on-screen location comprising information about the user interface element located at the on-screen location;

associating the semantic description of the user interface element located at the on-screen location with the hand position data; and

using the association of the semantic description of the user interface element located at the on-screen location with the hand position data to model user activity at the computing device.

17. The method of claim 1, comprising receiving typing force data comprising one or more real-time sensor inputs indicative of the amount of force or pressure applied by the user to a component of the computing device;

identifying an on-screen location corresponding to the typing force data;

obtaining a semantic description of a user interface element located at the on-screen location, the semantic description of the user interface element located at the on-screen location comprising information about the user interface element located at the on-screen location;

associating the semantic description of the user interface element located at the on-screen location with the typing force data; and

using the association of the semantic description of the user interface element located at the on-screen location with the typing force data to model user activity at the computing device.

18. A computing system to develop a semantic model of user attention to user interface elements of a plurality of software applications, the computing system comprising:

a display;

a sensor subsystem to obtain gaze-tracking data, the gaze-tracking data being indicative of a user'"'"'s gaze in relation to the display;

a framework embodied in one or more non-transitory machine-accessible media, the framework configured to, over time, during a plurality of user interactions involving a plurality of different software applications, generate semantic descriptions for each of the user interactions, wherein (i) at least a first one of the semantic descriptions relates to locations on the display corresponding to the gaze-tracking data; and

(ii) at least a second one of the semantic descriptions relates to application event data;

a model embodied in the one or more non-transitory machine-accessible media, the model configured to store at least (i) a semantic description relating to gaze-tracking data obtained during a user interaction involving one of the plurality of different computer applications and (ii) a semantic description relating to application event data generated as a result of another interaction involving another one of the plurality of different software applications and (iii) a recency or frequency of user attention to a user interface element, wherein the model is configured to initially store associations of semantic descriptions with the gaze-tracking data over time in a short-term memory portion of the model, and the model is configured to move at least a portion of the associations to a long-term memory portion of the model after a period of time; and

an inference engine embodied in the one or more non-transitory machine-accessible media, the inference engine to;

algorithmically draw an inference about a current user activity with the computing system based on the model; and

make the inference available for use by a computer-executable module of the computing system.

19. The computing system of claim 18, wherein the framework is configured to communicate with the plurality of software applications to obtain the semantic descriptions.

20. The computing system of claim 18, wherein the framework is configured to process the user interface elements to obtain the semantic descriptions.

21. A system for modeling user attention to user interface elements displayed by a plurality of software applications on a computing device, the system embodied in one or more non-transitory machine-accessible storage media, the system comprising:

a contextual model comprising data relating to a plurality of user interactions involving a plurality of different software applications, the contextual model comprising semantic descriptions of each of the plurality of user interactions, each of the semantic descriptions relating to one or more of;

a plurality of real-time sensor inputs received at the computing device, the real-time sensor inputs being indicative of a user'"'"'s gaze in relation to a display of the computing device;

locations on the display corresponding to the real-time sensor inputs;

the user interface elements displayed at the locations on the display corresponding to the real-time sensor inputs; and

application event data;

wherein the contextual model is configured to initially store associations of the semantic descriptions with gaze-tracking data over time in a short-term memory portion of the contextual model, and the contextual model is configured to move at least a portion of the associations to a long-term memory portion of the model after a period of time; and

an inference engine embodied in the one or more non-transitory machine-accessible media, the inference engine to;

algorithmically draw an inference about a current user activity with the computing device based on (i) a semantic description relating to gaze-tracking data obtained during a user interaction involving one of the plurality of different software applications and (ii) another semantic description relating to application event data generated as a result of another interaction involving another one of the plurality of different software applications and (iii) a recency or frequency of user attention to a user interface element; and

make the inference available for use by a computer-executable module of the computing device.

22. The system of claim 21, comprising a cognitive processing model to determine a degree of user cognitive processing of the user interface elements.

23. The system of claim 22, wherein the system is configured to determine whether to store the data based on the degree of user cognitive processing.

24. The system of claim 22, comprising a locomotive interaction model to analyze the time to perform user interactions with the computing device.

25. A method for modeling user activity with regard to a plurality of different software applications on a computing device, the method comprising, with the computing device, during a plurality of user interactions involving a plurality of different software applications over time:

receiving passive interaction data comprising one or more real-time sensor inputs indicative of a passive user interaction with the computing device, wherein the computing device is not configured to initiate any application events in response to the passive user interaction;

receiving active interaction data indicative of an active user interaction, wherein the computing device is configured to initiate at least one application event in response to the active user interaction;

generating a plurality of semantic descriptions relating to the passive interaction data and the active interaction data;

developing a model of user activity in relation to the computing device based on associations of the plurality of semantic descriptions with gaze-tracking data over time, initially storing the associations in a short-term memory portion of the model, and moving at least a portion of the associations to a long-term memory portion of the model after a period of time;

algorithmically drawing an inference about a current user activity with the computing device based on (i) a semantic description relating to gaze-tracking data obtained during a user interaction involving one of the plurality of different software applications and (ii) another semantic description relating to application event data generated as a result of another interaction involving another one of the plurality of different software applications and (iii) a recency or frequency of user attention to a user interface element; and

making the inference available for use by a computer-executable module of the computing device.

26. A method for modeling user activity with regard to a plurality of different software applications on a computing device, the method comprising, with a computing system comprising one or more computing devices:

generating a semantic description for each of a plurality of user interactions with the computing device performed over time with a plurality of different types of software applications, said plurality of user interactions comprising at least one passive user interaction and at least one active user interaction, and wherein;

for the at least one passive user interactions, said generating comprises;

receiving gaze-tracking data comprising one or more real-time sensor inputs indicative of a location of a user'"'"'s gaze in relation to a display of the computing device; and

generating a semantic description of the at least one passive user interaction based on information received from a one of the plurality of different software applications that is displaying content at the gaze location; and

for the at least one active user interactions, said generating comprises;

receiving, from at least one of the plurality of different software applications that is involved in said at least one active user interactions, a pre-defined textual description of an interactive function performed by the at least one of the plurality of software applications during the at least one active user interaction; and

generating a semantic description of the at least one active user interaction based on the pre-defined textual description;

developing a model of user activity in relation to the computing device based on associations of semantic descriptions with the gaze-tracking data over time, initially storing the associations in a short-term memory portion of the model, and moving at least a portion of the associations to a long-term memory portion of the model after a period of time;

algorithmically drawing an inference about a current user activity with the computing device based on the semantic description of the at least one passive user interaction and the semantic description of the at least one active user interaction and a recency or frequency of user attention to a user interface element; and

making the inference available for use by a computer-executable module of the computing system.