Method, apparatus, and system for facilitating cross-application searching and retrieval of content using a contextual user model

• US 9,152,222 B2
• Filed: 09/28/2012
• Issued: 10/06/2015
• Est. Priority Date: 05/17/2012
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Denker, Grit, Senanayake, Rukman
• Primary Examiner(s): Jacob, Ajith
• Application Number:
  US13/631,318
• Publication Number:
  US 20130311508A1
• Time in Patent Office: 1,103 Days
• Field of Search: 707/767
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2425 Iterative querying; Query formulation based on the results of a preceding query
• CPC Class Codes: G06F 16/332 Query formulation
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3349 Reuse of stored results of previous queries
• CPC Class Codes: G06F 16/90328 using search space presentation or visualization, e.g. category or range presentation and selection
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence

Abstract

A method, apparatus, and system for facilitating cross-application searching and retrieval of computer-stored content using a contextual user model includes using passive and/or active interaction data to formulate a user-specific search query. In some embodiments, inferences relating to the user'"'"'s current interaction context may be used to automatically retrieve relevant information for the user.

Claims

1. A content discovery module to retrieve content in an automated fashion at a computing device, comprising executable instructions embodied in one or more non-transitory machine-readable media, configured to cause a computing system comprising one or more computing devices to:

access a contextual user model, the contextual user model comprising;

gaze-tracking data comprising one or more real-time sensor inputs indicative of user gaze in relation to on-screen locations of a display of the computing device; and

semantic descriptions of user interface elements displayed at the on-screen locations corresponding to the gaze-tracking data, the semantic descriptions comprising information about the user interface elements;

generate an inference relating to a user-specific current interaction context based on the contextual user model;

formulate a search query including one or more aspects of the user-specific current interaction context;

execute the search query across one or more software applications; and

present one or more results of the search query at the computing device.

2. The content discovery module of claim 1, configured to determine, based on the gaze-tracking data, a plurality of recently-viewed user interface elements associated with a plurality of different software applications, determine search criteria based on one or more of the recently-viewed user interface elements, and formulate the search query using the search criteria.

3. The content discovery module of claim 2, wherein the recently-viewed user interface elements comprise an email message, a calendar appointment, an electronic document, or a web page.

4. The content discovery module of claim 1, configured to execute the search query across a plurality of networked computing devices.

5. The content discovery module of claim 1, configured to generate the inference based on a semantic description of a user interface element displayed at an on-screen location corresponding to the user gaze.

6. The content discovery module of claim 5, configured to incorporate at least a portion of the semantic description into the search query.

7. The content discovery module of claim 6, wherein the portion of the semantic description incorporated into the search query comprises a keyword or topic derived from the user interface element.

8. The content discovery module of claim 1, configured to generate the inference based on gaze-tracking data indicating a duration of user gaze at an on-screen location corresponding to a user interface element.

9. The content discovery module of claim 8 configured to incorporate into the search query at least a portion of a semantic description associated with the user interface element located at the on-screen location of user gaze.

10. The content discovery module of claim 1, configured to generate the inference based on gaze-tracking data indicating an intensity of user gaze at an on-screen location corresponding to a user interface element.

11. The content discovery module of claim 10, configured to incorporate into the search query at least a portion of a semantic description associated with the user interface element located at the on-screen location of user gaze.

12. The content discovery module of claim 1, configured to dynamically change the presentation of the results based on the user-specific current interaction context.

13. The content discovery module of claim 1, configured to present the results at an on-screen location corresponding to gaze-tracking data indicating a duration of user gaze.

14. The content discovery module of claim 1, configured to present the results at an on-screen location corresponding to gaze-tracking data indicating an intensity of user gaze.

15. The content discovery module of claim 1, configured to receive user feedback in response to a presented result and modify the presentation of results in response to the user feedback.

16. The content discovery module of claim 1, wherein the contextual user model comprises application event data from a plurality of software applications and semantic descriptions of the application event data, and the content discovery module is configured to incorporate into the search query at least a portion of a semantic description associated with the application event data.

17. A method for retrieving content in an automated fashion at a computing device, comprising:

maintaining a contextual user model, the contextual user model comprising;

gaze-tracking data comprising one or more real-time sensor inputs indicative of user gaze in relation to on-screen locations of a display of the computing device; and

semantic descriptions of user interface elements displayed at the on-screen locations corresponding to the gaze-tracking data, the semantic descriptions comprising information about the user interface elements;

generating an inference relating to a user-specific current interaction context based on the contextual user model;

formulating a search query including one or more aspects of the user-specific current interaction context;

executing the search query across one or more software applications; and

presenting one or more results of the search query at the computing device.

18. A computing system comprising:

a display;

a sensor subsystem to obtain gaze-tracking data, the gaze-tracking data being indicative of a user'"'"'s gaze in relation to the display;

one or more processors; and

one or more machine-readable media having stored therein a plurality of instructions that when executed by the processor cause the computing system to perform the method of claim 17.

19. A cognitive contextual search module to retrieve content in response to user input at a computing device, comprising executable instructions embodied in one or more non-transitory machine-readable media, configured to cause a computing system comprising one or more computing devices to:

receive a search request comprising a data characteristic and a user-specific interaction characteristic;

access a contextual user model, the contextual user model comprising;

gaze-tracking data comprising one or more real-time sensor inputs indicative of a person'"'"'s gaze in relation to on-screen locations of a display of the computing device; and

semantic descriptions of user interface elements displayed at the on-screen locations corresponding to the gaze-tracking data, the semantic descriptions comprising information about the user interface elements;

perform a search based on the search request and based on the contextual user model to obtain one or more search results; and

present the search results at the computing device.

20. The search module of claim 19, wherein the data characteristic relates to a software application and the interaction characteristic relates to a different software application.

21. The search module of claim 19, configured to interpret the search request using one or more of the semantic descriptions from the contextual user model.

22. The search module of claim 19, wherein the data characteristic comprises a keyword or meta data and the interaction characteristic comprises a reference to a prior user-specific interaction.

23. The search module of claim 22, configured to map the reference to a prior user-specific interaction to a semantic description contained in the contextual user model.

24. The search module of claim 19, configured to searchably index a plurality of content with prior interaction information from the contextual user model.

25. The search module of claim 24, wherein the searchably indexed content comprises content from a plurality of software applications.

26. The search module of claim 24, wherein the searchably indexed content comprises content located on a plurality of networked computing devices.

27. The search module of claim 19, configured to dynamically change the presentation of a search result based on the gaze-tracking data.

28. The search module of claim 19, configured to execute the search query across a plurality of software applications at the computing device.

29. The search module of claim 19, configured to receive user feedback in response to a presented search result and modify the presentation of the search result in response to the user feedback.

30. The search module of claim 19, wherein the contextual user model comprises application event data from a plurality of software applications and semantic descriptions of the application event data, and the search module is configured to interpret the search request based on a semantic description associated with the application event data.

31. A method for retrieving content in response to user input at a computing device, comprising:

receiving a search request comprising a data characteristic and a user-specific interaction characteristic;

maintaining a contextual user model, the contextual user model comprising;

gaze-tracking data comprising one or more real-time sensor inputs indicative of a person'"'"'s gaze in relation to on-screen locations of a display of the computing device; and

semantic descriptions of user interface elements displayed at the on-screen locations corresponding to the gaze-tracking data, the semantic descriptions comprising information about the user interface elements;

interpreting the search request based on the contextual user model;

developing a search query based on the interpreted search request;

executing the search query across one or more software applications to obtain one or more search results; and

presenting the search results at the computing device.

32. A computing system comprising:

a display;

a sensor subsystem to obtain gaze-tracking data, the gaze-tracking data being indicative of a user'"'"'s gaze in relation to the display;

one or more processors; and

one or more machine-readable media having stored therein a plurality of instructions that when executed by the processor cause the computing system to perform the method of claim 31.