Combined Activation for Natural User Interface Systems

• US 20120254810A1
• Filed: 03/31/2011
• Published: 10/04/2012
• Est. Priority Date: 03/31/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Heck, Larry Paul, Chinthakunta, Madhusudan, Mitby, David, Stifelman, Lisa
• Granted Patent: US 9,298,287 B2
• Time in Patent Office: Days
• US Class Current: 715/863
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A user interaction activation may be provided. A plurality of signals received from a user may be evaluated to determine whether the plurality of signals are associated with a visual display. If so, the plurality of signals may be translated into an agent action and a context associated with the visual display may be retrieved. The agent action may be performed according to the retrieved context and a result associated with the performed agent action may be displayed to the user.

Claims

1. A method for providing a user-interaction activation, the method comprising:

receiving a plurality of signals from a user;

determining whether the plurality of signals are associated with a visual display; and

in response to determining that the plurality of signals are associated with the visual display;

translating the plurality of signals into an agent action,

retrieving a context associated with the visual display,

performing the agent action according to the retrieved context, and

displaying a result associated with the performed agent action to the user.

2. The method of claim 1, wherein the visual display comprises at least one of the following:

a video and a still image.

3. The method of claim 1, wherein the visual display is associated with at least one of the following:

a television show, a movie, and a game.

4. The method of claim 1, wherein the plurality of signals comprises at least one of the following:

a keyword and an activation gesture.

5. The method of claim 1, wherein performing the agent action according to the retrieved context comprises narrowing a plurality of results to a subset of the plurality of results according to the retrieved context.

6. The method of claim 5, further comprising displaying the subset of the plurality of results to the user.

7. The method of claim 1, further comprising:

receiving a gesture from the user, wherein the gesture comprises at least one of the plurality of signals;

updating the retrieved context according to the gesture; and

performing the agent action according to the updated context.

8. The method of claim 7, wherein the gesture comprises a pointing received from the user.

9. The method of claim 8, wherein updating the retrieved context according to the pointing gesture comprises identifying an element of the visual display indicated by the pointing gesture.

10. A computer-readable medium which stores a set of instructions which when executed performs a method for providing a user-interaction activation, the method executed by the set of instructions comprising:

receiving a request from a user, wherein the request comprises a speech signal;

retrieving a context associated with a visual display;

identifying a gesture received from the user;

performing an action associated with the request according to the retrieved context and the identified gesture; and

providing a result associated with the performed action to the user.

11. The computer-readable medium of claim 10, wherein the natural language phrase comprises a spoken phrase.

12. The computer-readable medium of claim 10, wherein the visual display comprises at least one of the following a still image displayed to the user and a video image displayed to the user.

13. The computer-readable medium of claim 10, wherein the gesture and the speech signal are received from the user contemporaneously.

14. The computer-readable medium of claim 10, wherein the visual display comprises an image captured by a recording device associated with the user.

15. The computer-readable medium of claim 10, wherein the gesture comprises an activation gesture.

16. The computer-readable medium of claim 10, wherein the gesture is identified via a plurality of cameras.

17. The computer-readable medium of claim 10, wherein performing the action associated with the request according to the identified gesture comprises selecting a subset of the visual display to be associated with the request.

18. The computer-readable medium of claim 10, wherein the context associated with the visual display is retrieved from a plurality of metadata associated with the visual display.

19. The computer-readable medium of claim 10, wherein the context associated with the visual display is retrieved from a database.

20. A system for providing a user-interaction activation, the system comprising:

at least one camera;

a memory storage; and

a processing unit coupled to the memory storage and the camera, wherein the processing unit is operative to;

receive a plurality of contemporaneous signals from a user, wherein at least one first signal comprises a speech signal received via at least one microphone and at least one second signal comprises a gesture received via the at least one camera,

determine whether the plurality of signals are directed to the system, and

in response to determining that the plurality of signals are directed to the system;

retrieve a context associated with a visual display;

translate the plurality of signals into at least one agent action associated with the visual display, wherein the gesture comprises a pointing gesture operative to select a subset of the visual display;

perform the agent action according to the retrieved context and the gesture; and

display a result associated with the performed agent action to the user.