Device, method and system for monitoring, predicting, and accelerating interactions with a computing device

• US 9,046,917 B2
• Filed: 06/27/2012
• Issued: 06/02/2015
• Est. Priority Date: 05/17/2012
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: SRI International, Inc.
• Inventors: Senanayake, Rukman, Denker, Grit, Lincoln, Patrick D., Myers, Karen L., Gervasio, Melinda, Lee, Thomas J.
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Tran, Mai T
• Application Number:
  US13/534,155
• Publication Number:
  US 20130311411A1
• Time in Patent Office: 1,070 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2425 Iterative querying; Query formulation based on the results of a preceding query
• CPC Class Codes: G06F 16/332 Query formulation
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3349 Reuse of stored results of previous queries
• CPC Class Codes: G06F 16/90328 using search space presentation or visualization, e.g. category or range presentation and selection
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence

Abstract

A device, method and system for monitoring, predicting, and accelerating interactions with a computing device includes determining a current interaction context at a computing device based on interactions occurring at the computing device, predicting a number of potential subsequent interactions, speculatively executing at least some of the potential subsequent interactions, and presenting interactive representations of the speculatively-executed interactions at the computing device.

Claims

1. A method for accelerating user interaction with a computing device, comprising:

monitoring a current interaction context of the computing device, the current interaction context involving at least one current software application;

predicting a plurality of interactions with the computing device, including one or more inputs used in the predicted interactions, based on the current interaction context, at least one of the predicted interactions involving a different software application than the current software application;

speculatively executing each of the predicted interactions using a predicted input in response to the current interaction context, including executing the same predicted interaction multiple times using at least one different input each time; and

presenting, at the computing device, a user-selectable representation of each of the speculatively executed predicted interactions.

2. The method of claim 1, wherein the current interaction context relates to at least one interaction with the at least one current software application, comprising comparing the at least one interaction with a stored interaction model comprising data relating to a plurality of interactions with software applications.

3. The method of claim 2, wherein the interaction model comprises data relating to a plurality of interaction patterns comprising at least one of pre-defined interaction patterns, learned interaction patterns generated by a computerized algorithmic learning process and a plurality of user-specific interactions observed by the computing device.

4. The method of claim 1, wherein the plurality of predicted interactions includes an interaction that involves a software application and at least one parameter or argument and another interaction involving the same software application and at least one different parameter or argument.

5. The method of claim 1, wherein at least one of the predicted interactions comprises a sequence of multiple interactions that can be executed by the computing device in an automated fashion on behalf of a user.

6. The method of claim 1, comprising at least temporarily suspending the execution of a predicted interaction prior to completing the predicted interaction.

7. The method of claim 6, wherein the user-selectable representation of the at least one predicted interaction is selectable to continue executing the at least one temporarily suspended predicted interaction.

8. The method of claim 1, comprising speculatively executing a predicted interaction and speculatively executing another predicted interaction prior to completing execution of the predicted interaction.

9. The method of claim 1, comprising receiving a selection of one of the user-selectable representations and repeating the predicting, speculatively executing, and presenting based on the selection.

10. The method of claim 9, comprising updating an interaction model based on the selection, wherein the interaction model comprises data relating to a plurality of interactions.

11. The method of claim 1, comprising determining an amount of time associated with user performance of at least one of the predicted interactions and speculatively executing the at least one predicted interaction based on the amount of time.

12. The method of claim 1, comprising executing different predicted interactions using a different input in the execution of each different predicted interaction.

13. A computing device comprising:

at least one processor; and

computer circuitry coupled to the at least one processor, the computer circuitry being arranged to cause the at least one processor to;

monitor a current interaction context of the computing device, the current interaction context involving at least one current software application;

predict a plurality of interactions with the computing device, including one or more inputs used in the predicted interactions, based on the current interaction context, at least one of the predicted interactions involving a different software application than the current software application;

speculatively execute each of the predicted interactions using a predicted input in response to the current interaction context, including to execute the same predicted interaction multiple times using at least one different input each time; and

present at the computing device a user-selectable representation of each of the speculatively executed predicted interactions.

14. The computing device of claim 13, wherein the computing device is a personal handheld computing device.

15. At least one non-transitory machine-accessible storage medium comprising a plurality of instructions that in response to being executed result in a computing device:

monitoring a current interaction context of the computing device, the current interaction context involving at least one current software application;

predicting a plurality of interactions with the computing device, including one or more inputs used in the predicted interactions, based on the current interaction context, at least one of the predicted interactions involving a different software application than the current software application;

speculatively executing each of the predicted interactions using a predicted input in response to the current interaction context, including executing the same predicted interaction multiple times using at least one different input each time; and

presenting, at the computing device, a user-selectable representation of each of the speculatively executed predicted interactions.

16. A method for learning and accelerating user interaction with a computing device, comprising:

maintaining an interaction model comprising data relating to user interactions with the computing device in relation to one or more software applications;

monitoring a current interaction context of the computing device;

learning an interaction involving one or more software applications;

storing data relating to the learned interaction in the interaction model;

based on the current interaction context and the interaction model, predicting a plurality of interactions with the computing device, including one or more inputs used in the predicted interactions;

speculatively executing each of the predicted interactions using a predicted input in response to the current interaction context, including executing the same predicted interaction multiple times using at least one different input each time; and

presenting, at the computing device, a user-selectable representation of each of the speculatively executed predicted interactions.

17. The method of claim 16, comprising detecting a user-specified beginning of an interaction to be learned and detecting a user-specified end of the learned interaction.

18. The method of claim 16, comprising determining, in an automated fashion based on the current interaction context, a beginning and an end of the learned interaction.

19. The method of claim 16, comprising generalizing the learned interaction for applicability to interaction contexts that are different from the current interaction context.

20. A computing device, comprising:

a processor;

at least one optical sensor;

an output device; and

memory coupled to the processor, the memory having stored therein;

at least a portion of an interaction model, the interaction model comprising data relating to user interactions with the computing device in relation to one or more software applications; and

at least one computer-executable module configured to;

monitor a current interaction context of the computing device, the current interaction context defined at least in part by information derived from the at least one optical sensor;

based on the current interaction context, predict a plurality of interactions with the computing device, including one or more inputs used in the predicted interactions, each interaction involving at least one software application;

speculatively execute the predicted interactions in response to the current interaction context, including executing the same predicted interaction multiple times using at least one different input each time; and

present, at the output device, a user-selectable representation of each of the speculatively executed predicted interactions.

21. The computing device of claim 20, wherein the at least one computer-executable module is configured to modify the predicted interactions in response to information derived from the at least one optical sensor.

22. The computing device of claim 20, wherein the information derived from the at least one optical sensor relates to implicit behavior of a person with respect to the computing device, and the implicit behavior involves a portion of the person not explicitly interacting with the computing device.

23. The computing device of claim 20, wherein the at least one computer-executable module is configured to update the presentation of user-selectable representations based on the implicit behavior.

24. The computing device of claim 20, wherein the computing device is a personal handheld computing device.