METHOD, APPARATUS, AND SYSTEM FOR MODELING PASSIVE AND ACTIVE USER INTERACTIONS WITH A COMPUTER SYSTEM

• US 20130311924A1
• Filed: 09/28/2012
• Published: 11/21/2013
• Est. Priority Date: 05/17/2012
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: Rukman Senanayake
• Inventors: Denker, Grit, Senanayake, Rukman
• Granted Patent: US 9,152,221 B2
• Time in Patent Office: Days
• US Class Current: 715/771
• CPC Class Codes: G06F 16/2425 Iterative querying; Query formulation based on the results of a preceding query
• CPC Class Codes: G06F 16/332 Query formulation
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3349 Reuse of stored results of previous queries
• CPC Class Codes: G06F 16/90328 using search space presentation or visualization, e.g. category or range presentation and selection
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence

Abstract

A method, apparatus, and system for modeling user interactions with a computer system associates semantic descriptions of passive and active user interactions, which are meaningful at a user level, with application events and user interaction data as a user interacts with one or multiple software applications with a computing device, and uses those associations to build and maintain a user-specific contextual model. In some embodiments, the contextual models of multiple users are leveraged to form one or more collective contextual user models. Such models are useful in many different applications.

Claims

1. A method for modeling user activity with regard to a plurality of different software applications on a computing device, the method comprising, with the computing device:

receiving gaze-tracking data comprising one or more real-time sensor inputs indicative of a user'"'"'s gaze in relation to a display of the computing device;

identifying a user interface element displayed at a location on the display corresponding to the gaze-tracking data;

obtaining a semantic description of the user interface element, the semantic description comprising information about the user interface element;

associating the semantic description with the gaze-tracking data; and

using the association of the semantic description with the gaze-tracking data to model user activity at the computing device.

2. The method of claim 1, wherein the user interface element comprises one or more of text, graphic, video, interactive content, and an interactive control.

3. The method of claim 1, wherein the semantic description comprises functional information about the user interface element.

4. The method of claim 1, wherein the semantic description comprises information about content of the user interface element.

5. The method of claim 1, comprising identifying, from the plurality of different software applications, a software application associated with the user interface element.

6. The method of claim 5, comprising obtaining the semantic description from the software application.

7. The method of claim 1, comprising storing the associations of semantic descriptions with gaze-tracking data over time and developing a model of user activity in relation to the computing device based on the stored associations.

8. The method of claim 7, wherein the stored associations relate to content viewed at the computing device.

9. The method of claim 7, wherein the semantic descriptions relate to user interface elements displayed by the plurality of different software applications, and the method comprises developing a model of user activity with respect to the plurality of different software applications based on the stored associations.

10. The method of claim 1, wherein the gaze-tracking data comprises data relating to the duration of user attention to the user interface element and the method comprises generating an inference based on the data relating to the duration of user attention to the user interface element.

11. The method of claim 10, wherein the inference relates to user cognitive processing of content of the user interface element.

12. The method of claim 1, wherein the gaze-tracking data comprises data relating to the recency or frequency of user attention to the user interface element and the method comprises generating an inference based on the recency or frequency of user attention to the user interface element.

13. The method of claim 12, comprising developing a model of user activity in relation to the computing device based on associations of semantic descriptions with gaze-tracking data over time, initially storing the associations in a short-term memory portion of the model, and moving the associations to a long-term memory portion of the model after a period of time.

14. The method of claim 13, comprising determining the period of time based on the recency of user attention to the user interface element, the time of day, demographic information, location, or context.

15. The method of claim 1, comprising receiving, from one or more of the plurality of different software applications, data relating to user interaction with the user interface element and associating the interaction data with the gaze-tracking data.

16. The method of claim 1, comprising associating a cognitive processing parameter relating to the gaze-tracking data, wherein the cognitive processing parameter indicates a degree of user cognitive processing of content of the user interface element.

17. The method of claim 1, comprising receiving application event data from a software application at the computing device, obtaining a semantic description of the application event, the semantic description comprising information about the application event, and using the association of the semantic description with the application event data to model user activity at the computing device.

18. The method of claim 1, comprising receiving hand position data comprising one or more real-time sensor inputs indicative of the proximity of at least a portion of a user'"'"'s hand in relation to a component of the computing device;

identifying an on-screen location corresponding to the hand position data;

obtaining a semantic description of a user interface element located at the on-screen location, the semantic description comprising information about the user interface element;

associating the semantic description with the hand position data; and

using the association of the semantic description with the hand position data to model user activity at the computing device.

19. The method of claim 1, comprising receiving typing force data comprising one or more real-time sensor inputs indicative of the amount of force or pressure applied by a user to a component of the computing device;

identifying an on-screen location corresponding to the typing force data;

obtaining a semantic description of a user interface element located at the on-screen location, the semantic description comprising information about the user interface element;

associating the semantic description with the typing force data; and

using the association of the semantic description with the typing force data to model user activity at the computing device.

20. A computing system to develop a semantic model of user attention to user interface elements of a plurality of software applications, the computing system comprising:

a display;

a sensor subsystem to obtain gaze-tracking data, the gaze-tracking data being indicative of a user'"'"'s gaze in relation to the display;

a framework embodied in one or more machine-accessible media, the framework configured to, over time;

determine locations on the display corresponding to the gaze-tracking data;

identify user interface elements displayed by the software applications at each of the locations corresponding to the gaze-tracking data;

obtain a semantic description of each of the user interface elements, each of the semantic descriptions comprising information about the corresponding user interface element; and

associate the semantic descriptions with the gaze-tracking data; and

a model embodied in one or more machine-accessible media, the model configured to store data relating to associations of the semantic descriptions with the gaze-tracking data.

21. The computing system of claim 20, wherein the framework is configured to communicate with the plurality of software applications to obtain the semantic descriptions.

22. The computing system of claim 20, wherein the framework is configured to process the user interface elements to obtain the semantic descriptions.

23. A system for modeling user attention to user interface elements displayed by a plurality of software applications on a computing device, the system embodied in one or more machine-accessible storage media, the system comprising:

a contextual model comprising data relating to;

a plurality of real-time sensor inputs received at a computing device, the real-time sensor inputs being indicative of a user'"'"'s gaze in relation to a display of the computing device;

locations on the display corresponding to the real-time sensor inputs; and

user interface elements displayed at the locations corresponding to the real-time sensor inputs; and

a framework configured to;

derive gaze-tracking data from the real-time sensor inputs, the gaze-tracking data indicating an aspect of user attention to the user interface elements;

determine semantic descriptions of the user interface elements, each of the semantic descriptions comprising information about the corresponding user interface element;

associate the semantic descriptions with the gaze-tracking data; and

store data relating to the associations of the semantic descriptions with the gaze-tracking data in the contextual model.

24. The system of claim 23, comprising a cognitive processing model to determine a degree of user cognitive processing of the user interface elements.

25. The system of claim 24, wherein the framework determines whether to store the data based on the degree of user cognitive processing.

26. The system of claim 24, comprising a locomotive interaction model to analyze the time to perform user interactions with the computing device.

27. A method for modeling user activity with regard to a plurality of different software applications on a computing device, the method comprising, with the computing device:

receiving passive interaction data comprising one or more real-time sensor inputs indicative of a passive user interaction with the computing device, the passive user interaction being an interaction that does not result in an application event;

receiving active interaction data indicative of an active user interaction, the active user interaction being an interaction that results in an application event;

identifying user interface elements displayed at on-screen locations of the display corresponding to the passive and active interaction data;

obtaining semantic descriptions of the user interface elements, the semantic descriptions comprising information about the user interface elements;

associating the semantic descriptions with the corresponding passive and active interaction data; and

using the associations of semantic descriptions with the passive and active interaction data to model user activity at the computing device.