Systems and methods for switching processing modes using gestures

• US 9,436,287 B2
• Filed: 03/15/2013
• Issued: 09/06/2016
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Ton, Phuong L., Hildreth, Evan R., Bernarte, Joel S., Arellano, Suzana
• Primary Examiner(s): ULRICH, NICHOLAS S
• Application Number:
  US13/835,234
• Publication Number:
  US 20140278441A1
• Time in Patent Office: 1,271 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 21/00 Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility G10L19/00 takes precedence

Abstract

Systems and methods for switching between voice dictation modes using a gesture are provided so that an alternate meaning to a dictated word may be applied. The provided systems and methods time stamp detected gestures and detected words from the voice dictation and compare the time stamp at which a gesture is detected to the time stamp at which a word is detected. When it is determined that a time stamp of a gesture approximately matches a time stamp of a word, the word may be processed to have an alternate meaning, such as a command, punctuation, or action.

Claims

1. A system for processing detected audio waveforms in a first mode and a second mode, comprising:

a microphone configured for detecting one or more audio waveforms;

a gesture detection sensor configured to detect one or more touchless gestures corresponding to different processing modes;

a system clock configured to provide time stamps for the one or more audio waveforms detected by the microphone and time stamps for the one or more touchless gestures detected by the gesture detection sensor, the time stamps for the one or more touchless gestures indicating a starting time and an ending time of each detected touchless gesture;

a buffer configured to store the one or more audio waveforms and information regarding the one or more touchless gestures; and

one or more processors configured to enter the second mode and selectively process a detected audio waveform stored in the buffer in the second mode when a time stamp of the detected audio waveform is between a first time stamp of a first touchless gesture corresponding to the second mode detected by the gesture detection sensor and a second time stamp associated with one of the one or more touchless gestures.

2. The system of claim 1, wherein the one or more processors are further configured to process a detected audio waveform in the first mode when a time stamp of the detected audio waveform is before the first time stamp or after the second time stamp.

3. The system of claim 1, further comprising a memory storing first meanings for detected audio waveforms processed in the first mode and second meanings for some detected audio waveforms processed in the second mode.

4. The system of claim 3, wherein the second meanings comprise at least one of a command, a punctuation mark, or an action.

5. The system of claim 4, wherein the first meanings comprise text associated with the detected audio waveforms.

6. The system of claim 1, wherein the gesture detection sensor comprises at least one of a camera, an ultrasound transducer, a shadow detector, or an infrared sensor.

7. The system of claim 1, further comprising a display, the display configured to display feedback indicating when the detected audio waveform is being processed in the second mode.

8. The system of claim 1,

wherein each of the one or more touchless gestures corresponds to a respective mode of a plurality of modes, and

wherein the one or more processors are configured to selectively process an audio waveform detected by the microphone in one mode of the plurality of modes when:

a time stamp of the detected audio waveform is between a time stamp corresponding to a start of one touchless gesture of the one or more touchless gestures corresponding to the one mode detected by the gesture detection sensor and a time stamp corresponding to an end of the one touchless gesture;

or

a time stamp of the detected audio waveform is between the time stamp corresponding to the start of the one touchless gesture and a time stamp corresponding to a start of a next detected touchless gesture.

9. The system of claim 8, wherein the plurality of modes includes at least one of a correction mode, a command mode, or a translation mode.

10. The system of claim 1, wherein the detected audio waveforms are provided to the one or more processors from the buffer for processing based on the one or more touchless gestures detected by the gesture detection sensor.

11. The system of claim 1, wherein the one or more processors are configured to enable a gesture detection procedure using the gesture detection sensor when the microphone detects an audio waveform.

12. The system of claim 11, wherein the gesture detection sensor is inactive until the detected audio waveform is determined to be a human voice.

13. The system of claim 1, wherein the second mode includes one of a correction mode, a command mode, or a translation mode.

14. The system of claim 1,

wherein the first time stamp of the first touchless gesture corresponds to a start of the first touchless gesture;

and

wherein the second time stamp corresponds to an end of the first touchless gesture.

15. The system of claim 1,

wherein the first time stamp of the first touchless gesture corresponds to a start of the first touchless gesture;

and

wherein the second time stamp corresponds to a start of a next detected touchless gesture corresponding to the first mode.

16. The system of claim 1, wherein the time stamps for the one or more audio waveforms are captured at pauses between words.

17. The system of claim 1, wherein the time stamps for the one or more audio waveforms are captured at a beginning and an end of a string including multiple words, and time stamps for individual words are determined by interpolating the time stamps captured at the beginning and the end of the string.

18. A method for processing detected audio waveforms in a first mode and a second mode, comprising:

detecting, by a microphone, one or more audio waveforms;

time stamping the detected one or more audio waveforms;

detecting, by a gesture detection sensor, one or more touchless gestures corresponding to one or more modes;

time stamping a starting time and an ending time of each of the detected one or more touchless gestures;

storing, in a buffer, the one or more audio waveforms and information regarding the one or more touchless gestures; and

entering the second mode and selectively processing, by one or more processors coupled to the microphone, an audio waveform of the one or more audio waveforms in the second mode when a time stamp of the audio waveform is between a first time stamp of a first touchless gesture corresponding to the second mode detected by the gesture detection sensor and a second time stamp associated with one of the one or more touchless gestures.

19. The method of claim 18,

wherein the first time stamp of the first touchless gesture corresponds to a start of the first touchless gesture;

and

wherein the second time stamp corresponds to an end of the first touchless gesture or a start of a next detected touchless gesture corresponding to the first mode.

20. The method of claim 18, further comprising processing the audio waveform in the first mode when a time stamp of the audio waveform is before the first time stamp or after the second time stamp.

21. The method of claim 18, wherein processing the audio waveform in the second mode comprises correlating the audio waveform with a meaning that is different from a first meaning corresponding to the first mode.

22. The method of claim 21, wherein the meaning that is different from the first meaning comprises at least one of a command, a punctuation mark, or an action.

23. The method of claim 22, wherein the first meaning comprises text associated with the audio waveform.

24. The method of claim 18, wherein detecting the one or more touchless gestures comprises detecting if the one or more touchless gestures are performed using at least one of a camera, an ultrasound transducer, a shadow detector, or an infrared sensor.

25. The method of claim 18, wherein processing the audio waveform in the second mode comprises generating audio or visual feedback indicating that the audio waveform is being processed in the second mode.

26. The method of claim 18,

wherein each of the one or more touchless gestures corresponds to a respective mode of the one or more modes, and

wherein selectively processing comprises selectively processing the audio waveform in a mode of the one or more modes corresponding to one touchless gesture of the one or more touchless gestures when:

a time stamp of the audio waveform is between a time stamp corresponding to a start of the one touchless gesture detected by the gesture detection sensor and a time stamp corresponding to an end of the one touchless gesture;

or

a time stamp of the audio waveform is between the time stamp corresponding to the start of the one touchless gesture and a time stamp corresponding to a start of a next detected touchless gesture of the one or more touchless gestures.

27. The method of claim 18, further comprising:

providing the one or more audio waveforms from the buffer when the one or more touchless gestures are detected.

28. The method of claim 18, further comprising enabling a gesture detection procedure in response to the microphone detecting an audio waveform.

29. A non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform a method for processing detected audio waveforms in a first mode and a second mode, the method comprising:

detecting one or more audio waveforms;

time stamping the detected one or more audio waveforms;

detecting one or more touchless gestures corresponding to one or more modes;

time stamping a starting time and an ending time of each of the detected one or more touchless gestures;

storing, in a buffer, the one or more audio waveforms and information regarding the one or more gestures; and

entering the second mode and selectively processing an audio waveform of the one or more audio waveforms in the second mode when a time stamp of the audio waveform is between a first time stamp of a first touchless gesture of the one or more touchless gestures corresponding to the second mode detected by the gesture detection sensor and a second time stamp associated with one of the one or more touchless gestures.

30. A system for processing detected audio waveforms in a first mode and a second mode, comprising:

means for detecting one or more audio waveforms and time stamping the one or more audio waveforms;

means for detecting one or more touchless gestures corresponding to one or more modes and time stamping a starting time and an ending time of each of the detected one or more touchless gestures;

means for storing the one or more audio waveforms and information regarding the one or more gestures; and

means for entering the second mode and selectively processing an audio waveform from the one or more audio waveforms in the second mode when a time stamp of the audio waveform is between a first time stamp of a first detected touchless gesture of the one or more touchless gestures corresponding to the second mode and a second time stamp associated with one of the one or more touchless gestures.

31. The system of claim 30,

wherein the first time stamp of the first detected touchless gesture corresponds to a start of the first detected touchless gesture;

and

wherein the second time stamp corresponds to an end of the first detected touchless gesture or a start of a next detected touchless gesture corresponding to the first mode.

32. The system of claim 30, wherein the means for selectively processing processes the audio waveform in the first mode when a time stamp of the audio waveform is before the first time stamp or after the second time stamp.

33. The system of claim 30, wherein the means for selectively processing processes the audio waveform in the second mode by correlating the audio waveform with a second meaning that is different from a first meaning corresponding to the first mode.

34. The system of claim 33, wherein the second meaning that is different from the first meaning comprises at least one of a command, a punctuation mark, or an action.

35. The system of claim 34, wherein the first meaning comprises text associated with the audio waveform input.

36. The system of claim 30, wherein the means for detecting the one or more touchless gestures comprises at least one of a camera, an ultrasound transducer, a shadow detector, or an infrared sensor.

37. The system of claim 30, wherein the means for selectively processing generates audio or visual feedback indicating that the audio waveform is being processed in the second mode.

38. The system of claim 30,

wherein each of the one or more touchless gestures corresponding to a respective mode of the one or more modes, and

wherein the means for selectively processing comprises selectively processing the audio waveform in a mode of the one or more modes corresponding to one touchless gesture of the one or more touchless gestures when:

a time stamp of the audio waveform is between a time stamp corresponding to a start of the one touchless gesture detected by the gesture detection sensor and a time stamp corresponding to an end of the one touchless gesture;

or

a time stamp of the audio waveform is between the time stamp corresponding to the start of the one touchless gesture and a time stamp corresponding to a start of a next detected touchless gesture of the one or more touchless gestures.

39. The system of claim 30,

wherein the means for storing the audio waveforms provides the stored audio waveforms to the means for selectively processing when the means for detecting the one or more touchless gestures detects the one or more touchless gestures.