COMPOUND GESTURE-SPEECH COMMANDS

• US 20130027296A1
• Filed: 10/06/2012
• Published: 01/31/2013
• Est. Priority Date: 06/18/2010
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Flaks, Jason S., Klein, Christian, Vassigh, Ali M., Soemo, Thomas M., Larco, Vanessa
• Application Number:
  US13/646,692
• Publication Number:
  US 20130027296A1
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/226 using non-speech characteristics

Abstract

A multimedia entertainment system combines both gestures and voice commands to provide an enhanced control scheme. A user'"'"'s body position or motion may be recognized as a gesture, and may be used to provide context to recognize user generated sounds, such as speech input. Likewise, speech input may be recognized as a voice command, and may be used to provide context to recognize a body position or motion as a gesture. Weights may be assigned to the inputs to facilitate processing. When a gesture is recognized, a limited set of voice commands associated with the recognized gesture are loaded for use. Further, additional sets of voice commands may be structured in a hierarchical manner such that speaking a voice command from one set of voice commands leads to the system loading a next set of voice commands.

Claims

1. A method for controlling a computing system using a voice commands, comprising:

accessing multiple depth images from a depth sensor system;

recognizing a gesture from the multiple depth images;

in response to recognizing a gesture, choosing a subset of a set of sound commands based on the recognized gesture, the set of sound commands includes multiple subsets, each subset is associated with one or more gestures and sound command recognition data for the respective subset;

receiving sound input;

recognizing a sound command from the chosen subset based on the sound input; and

performing an action in response to the recognized sound command.

2. The method of claim 1, wherein:

the depth images include a two-dimensional pixel area of a captured scene, the pixels in the two-dimensional pixel area represent depth values of one or more objects in the scene captured by the depth sensor system.

3. The method of claim 1, further comprising:

loading sound command recognition data for the chosen subset of sound commands.

4. The method of claim 1, wherein:

the sound input is received from a microphone.

5. The method of claim 1, wherein:

the recognizing includes attempting to match the sound input to sound commands in the chosen subset and not to sound commands in other subsets of the set of sound commands.

6. The method of claim 1, further comprising:

displaying the chosen subset of sound commands in response to the recognizing the gesture.

7. The method of claim 6, wherein:

the recognized gesture selects a displayed object; and

the chosen subset of sound commands is displayed proximate to the selected displayed object.

8. The method of claim 1, further comprising:

after performing the action in response to the recognized sound command, loading additional sound command recognition data for a related subset of the chosen subset of sound commands;

receiving a further sound command;

recognizing the further sound command using the loaded additional sound command recognition data; and

performing an action in response to the recognized further sound command.

9. The method of claim 1, further comprising:

changing the state of the computing system based on the recognized gesture; and

providing a hierarchical subset of sound commands related to the changed state of the computing system.

10. The method of claim 1, wherein the depth images and the sound command are received substantially simultaneously.

11. The method of claim 1, wherein:

the depth images include a two-dimensional pixel area of a captured scene, each pixel in the two-dimensional pixel area represents a depth value of one or more objects in the scene captured by the depth sensor system;

the method further comprises displaying the chosen subset of sound commands in response to the recognizing the gesture;

the sound input is received after displaying the chosen subset of sound commands; and

the recognizing includes attempting to match the sound input to sound commands in the chosen subset and not to sound commands in other subsets of the set of sound commands.

12. A computing system, comprising:

a monitor for displaying multimedia content;

a depth sensor for capturing depth images;

a microphone for capturing sounds; and

a processor in communication with the depth sensor, the microphone and the monitor;

the processor communicates with the monitor to display an object, the processor receives multiple depth images from the depth sensor and recognizes a gesture from the multiple depth images, the processor chooses a subset of a set of sound commands based on and in response to the recognized gesture, the set of sound commands includes multiple subsets, the processor loads sound command recognition data for the chosen subset of sound commands in response to the recognized gesture, the processor receives sound input from the microphone and recognizes a sound command from the chosen subset based on the sound input and the loaded sound command recognition data without searching all of the set of sound commands, processor performs an action in response to the recognized sound command.

13. The system of claim 12, wherein:

the subset of sound commands is organized in one or more hierarchical levels, wherein each hierarchical level is only loaded as necessary to process a gesture.

14. The system of claim 13, wherein:

a subsequent hierarchical level of sound commands is loaded and displayed only after a sound command from a previous hierarchical level has been recognized and the desired action associated with the sound command taken.

15. The system of claim 12, wherein:

the gesture is used by the processor to facilitate recognition of the sound command, and the sound command is used by the processor to facilitate recognition of the gesture.

16. The system of claim 12, wherein:

the processor recognizes the gesture by attempting to match the sound input to sound commands in the chosen subset and not to sound commands in other subsets of the set of sound commands.

17. The system of claim 12, wherein:

the processor displays the chosen subset of sound commands in response to the recognizing the gesture, and receives the sound input after displaying the chosen subset of sound commands.

18. A processor readable storage device having instructions thereon, the instructions for programming one or more processors to perform a method, comprising:

receiving sound input;

accessing multiple depth images from a depth sensor;

recognizing a gesture from the multiple depth images based on tracking motion of an object moving in a scene captured in the depth images;

in response to recognizing the gesture, choosing a subset of a set of sound commands based on the recognized gesture; and

identifying a sound command using a recognition process that is limited to attempting to match the sound input to sound commands of the chosen subset and not to sound commands of the set of sound commands not in the chosen subset.

19. The processor readable storage device of claim 18, wherein:

the method further comprises displaying the chosen subset of sound commands in response to the recognizing the gesture, the sound input is received after displaying the chosen subset of sound commands.

20. The processor readable storage device of claim 18, wherein:

the recognizing the gesture from the multiple depth images based on tracking motion of the object comprises performing skeletal tracking of the object based on the depth images and determining a gesture being performed based on the skeletal tracking.