Systems and methods of performing speech recognition using gestures

• US 8,321,219 B2
• Filed: 09/25/2008
• Issued: 11/27/2012
• Est. Priority Date: 10/05/2007
• Status: Active Grant

• Current Assignee: Sensory Incorporated
• Original Assignee: Sensory Incorporated
• Inventors: Mozer, Todd F.
• Primary Examiner(s): RIDER, JUSTIN W
• Application Number:
  US12/238,287
• Publication Number:
  US 20090150160A1
• Time in Patent Office: 1,524 Days
• Field of Search: 704/236
• US Class Current: 704/236
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

Embodiments of the present invention improve methods of performing speech recognition using human gestures. In one embodiment, the present invention includes a speech recognition method comprising detecting a gesture, selecting a first recognition set based on the gesture, receiving a speech input signal, and recognizing the speech input signal in the context of the first recognition set.

Claims

1. A speech recognition method comprising:

initiating, by a computer system, a state of a computer program;

selecting, by the computer system, a state recognition set based on the state of the computer program;

detecting, by the computer system a gesture of at least one part of a human body;

selecting, by the computer system, a first recognition set based on the gesture, the first recognition set being a subset of the state recognition set;

receiving, by the computer system, a speech input signal; and

recognizing, by the computer system, the speech input signal in the context of the first recognition set, the recognizing resulting in a recognition result.

2. The method of claim 1 wherein the gesture is a hand gesture, wherein the step of detecting the gesture includes detecting a portion of a human hand.

3. The method of claim 1 wherein the gesture includes motion.

4. The method of claim 1 wherein the step of detecting includes the use of infrared light emissions.

5. The method of claim 1 wherein the gesture is from a set of gestures corresponding to a plurality of recognition sets,

wherein each recognition set corresponds to at least one gesture in the set of gestures.

6. The method of claim 1 wherein the recognizing includes using the first recognition set to weight a set of likelihoods, wherein the first recognition set includes segments of speech.

7. The method of claim 1 further comprising reducing a level of sound generation in response to the step of detecting the gesture.

8. The method of claim 1 further comprising:

initiating a new state of the computer program based on the recognition result; and

selecting a new state recognition set based on the new state of the computer program.

9. The method of claim 8 further comprising providing a visual cue corresponding to the new state of the computer program.

10. The method of claim 9 further comprising detecting a second gesture.

11. The method of claim 10 further comprising adjusting a parameter of the computer program corresponding to the second gesture.

12. The method of claim 1 further comprising providing a voice prompt corresponding to the gesture.

13. A speech recognition system comprising:

a controller;

a gesture detector coupled to provide signals to the controller, the signals corresponding to a gesture;

a speech recognizer having access to a plurality of recognition sets, the speech recognizer coupled to provide recognition results to the controller;

an audio user interface coupled to receive a speech input and coupled to provide audio signals to the speech recognizer, the audio signals corresponding to the speech input; and

a plurality of output devices coupled to the controller,

wherein a recognition set is selected from the plurality of recognition sets based on the gesture,

wherein the recognition results are generated based on the speech input and the selected recognition set,

wherein each recognition set in the plurality of recognition sets is associated with an output device in the plurality of output devices, and

wherein the controller provides signals to the output device associated with the selected recognition set, the signals corresponding to the recognition results.

14. The system of claim 13 wherein the gesture detector includes a plurality of infrared detectors arranged to provide signals corresponding to the gesture.

15. The system of claim 13 wherein at least a portion of the plurality of output devices are coupled through a wireless network.

16. The system of claim 13 wherein at least one of the output devices of the plurality of output devices is a speech synthesizer that provides a verbal prompt corresponding to the gesture.

17. A speech recognition method comprising:

initiating, by a computer system, a state of a computer program;

selecting, by the computer system, a state recognition set based on the state of the computer program;

detecting, by the computer system a gesture of at least one part of a human body;

receiving, by the computer system, a speech input signal; and

recognizing, by the computer system, the speech input signal in the context of a recognition set, the recognition set being a subset of the state recognition set, the recognizing resulting in a first recognition result;

weighting, by the computer system, the first recognition result according to a function associated with the gesture; and

determining, by the computer system, a second recognition result based on the weighting.

18. The method of claim 17

wherein the computer program includes the function associated with the gesture.

19. The method of claim 17 wherein the gesture is a hand gesture, wherein the step of detecting the gesture includes detecting a portion of a human hand.

20. The method of claim 17 wherein the gesture includes motion.

21. The method of claim 17 wherein the step of detecting includes the use of infrared light emissions.

22. The method of claim 17 wherein the gesture is from a set of gestures corresponding to a plurality of functions,

wherein each function corresponds to at least one gesture in the set of gestures.

23. A system comprising:

an automobile; and

a speech recognition system in a cabin of said automobile, the speech recognition system comprising,

a controller,

a gesture detector coupled to provide signals to the controller, the signals corresponding to a gesture,

a speech recognizer having access to a plurality of recognition sets, the speech recognizer coupled to provide recognition results to the controller,

an audio user interface coupled to receive a speech input and coupled to provide audio signals to the speech recognizer, the audio signals corresponding to the speech input, and

a plurality of output devices coupled to the controller,

wherein a recognition set is selected from the plurality of recognition sets based on the gesture,

wherein the recognition results are generated based on the speech input and the selected recognition set,

wherein each recognition set in the plurality of recognition sets is associated with an output device in the plurality of output devices, and

wherein the controller provides signals to the output device associated with the selected recognition set, the signals corresponding to the recognition results.