SYSTEM AND METHOD FOR CONTINUOUS MULTIMODAL SPEECH AND GESTURE INTERACTION

• US 20130144629A1
• Filed: 12/01/2011
• Published: 06/06/2013
• Est. Priority Date: 12/01/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: JOHNSTON, Michael, OZKAN, Derya
• Granted Patent: US 9,152,376 B2
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

Disclosed herein are systems, methods, and non-transitory computer-readable storage media for processing multimodal input. A system configured to practice the method continuously monitors an audio stream associated with a gesture input stream, and detects a speech event in the audio stream. Then the system identifies a temporal window associated with a time of the speech event, and analyzes data from the gesture input stream within the temporal window to identify a gesture event. The system processes the speech event and the gesture event to produce a multimodal command. The gesture in the gesture input stream can be directed to a display, but is remote from the display. The system can analyze the data from the gesture input stream by calculating an average of gesture coordinates within the temporal window.

Claims

1. A method comprising:

continuously monitoring an audio stream associated with a gesture input stream;

detecting a speech event in the audio stream;

identifying a temporal window associated with a time of the speech event;

analyzing, via a processor, data from the gesture input stream within the temporal window to identify a gesture event; and

processing the speech event and the gesture event to produce a multimodal command.

2. The method of claim 1, wherein a gesture in the gesture input stream is directed to a display, but is remote from the display.

3. The method of claim 1, wherein analyzing the data from the gesture input stream further comprises calculating an average of gesture coordinates within the temporal window.

4. The method of claim 1, wherein processing the speech event and the gesture event further comprises:

identifying a speech command from the speech event;

identifying parameters from the gesture event; and

applying the parameters to the speech command.

5. The method of claim 1, wherein a gesture filtering module focuses the temporal window based on timing of specific words in the speech event.

6. The method of claim 1, further comprising executing the multimodal command.

7. The method of claim 1, wherein at least one of a length and a position of the temporal window is based on a type of the speech event.

8. The method of claim 1, wherein the speech event is detected in the audio stream without an explicit user activation via one of a button press and a touch gesture.

9. The method of claim 1, wherein the gesture input stream comprises input from at least one of a motion detector, a motion capture system, a camera, and an infrared camera.

10. The method of claim 1, wherein the audio stream comprises input from at least one of a microphone and an array of microphones.

11. The method of claim 1, wherein identifying the temporal window comprises modifying the temporal window based on one of the gesture event and any other data to yield a modified temporal window.

16. The system of claim 1, wherein a gesture filtering module focuses the temporal window based on timing of specific words in the speech event.

12. A system comprising:

a processor;

a memory having stored therein instructions for controlling the processor to perform steps comprising;

monitoring an audio stream associated with a gesture input stream;

detecting a speech event in the audio stream;

identifying a temporal window associated with a time of the speech event;

analyzing, via a processor, data from the gesture input stream within the temporal window to identify a gesture event; and

processing the speech event and the gesture event to produce a multimodal command.

13. The system of claim 12, wherein a gesture in the gesture input stream is directed to a display, but is remote from the display.

14. The system of claim 12, wherein analyzing the data from the gesture input stream further comprises calculating an average of gesture coordinates within the temporal window.

15. The system of claim 12, wherein processing the speech event and the gesture event further comprises:

identifying a speech command from the speech event;

identifying parameters from the gesture event; and

applying the parameters to the speech command.

17. A non-transitory computer-readable storage medium having stored therein instructions which, when executed by a computing device, cause the computing device to perform steps comprising:

continuously monitoring an audio stream associated with a gesture input stream;

detecting a speech event in the audio stream;

identifying a temporal window associated with a time of the speech event;

analyzing, via a processor, data from the gesture input stream within the temporal window to identify a gesture event; and

processing the speech event and the gesture event to produce a multimodal command.

18. The non-transitory computer-readable storage medium of claim 17, wherein the speech event is detected in the audio stream without an explicit user activation via one of a button press and a touch gesture.

19. The non-transitory computer-readable storage medium of claim 17, wherein the gesture input stream comprises input from at least one of a motion detector, a motion capture system, a camera, and an infrared camera.

20. The non-transitory computer-readable storage medium of claim 17, wherein the audio stream comprises input from at least one of a microphone and an array of microphones.