Multisensory speech detection

• US 9,009,053 B2
• Filed: 11/10/2009
• Issued: 04/14/2015
• Est. Priority Date: 11/10/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Burke, Dave, Lebeau, Michael J., Gianno, Konrad, Kristjansson, Trausti, Jitkoff, John Nicholas, Senior, Andrew W.
• Primary Examiner(s): Feliciano, Eliseo Ramos
• Assistant Examiner(s): STIFTER JR, TERENCE E
• Application Number:
  US12/615,583
• Publication Number:
  US 20100121636A1
• Time in Patent Office: 1,981 Days
• Field of Search: 704/233
• US Class Current: 704/270
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/10 using distance or distortion measures between unknown speech and reference templates
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 25/21 the extracted parameters being power information
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04M 2250/74 with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00
• CPC Class Codes: H04R 1/08 Mouthpieces; Microphones; Attachments therefor
• CPC Class Codes: H04W 4/026 using orientation information, e.g. compass

Abstract

A computer-implemented method of multisensory speech detection is disclosed. The method comprises determining an orientation of a mobile device and determining an operating mode of the mobile device based on the orientation of the mobile device. The method further includes identifying speech detection parameters that specify when speech detection begins or ends based on the determined operating mode and detecting speech from a user of the mobile device based on the speech detection parameters.

Claims

1. A computer-implemented method, comprising:

determining a first orientation of a mobile device;

determining a first speech energy threshold based on the first orientation of the mobile device,

the first speech energy threshold being usable to determine that a user of the mobile device has stopped speaking;

detecting, using one or more processors, that the user of the mobile device has stopped speaking based on a comparison of first detected auditory information to the first speech energy threshold;

determining a second orientation of the mobile device, the second orientation of the mobile device being distinct from the first orientation of the mobile device;

determining a second speech energy threshold based on the second orientation of the mobile device,

the second speech energy threshold being usable to determine that the user of the mobile device has stopped speaking, the second speech energy threshold being distinct from the first speech energy threshold;

detecting, using the one or more processors, that the user of the mobile device has stopped speaking based on a comparison of second detected auditory information to the second speech energy threshold.

2. The method of claim 1, wherein determining the first orientation of the mobile device comprises detecting an angle of the mobile device.

3. The method of claim 1, wherein determining the first orientation of the mobile device comprises detecting a proximity of the mobile device to the user of the mobile device.

4. The method of claim 1, further comprising indicating to the user of the mobile device that speech detection has started or ended.

5. The method of claim 4, wherein indicating to the user of the mobile device that speech detection has started or ended comprises a visual or auditory notification.

6. The computer-implemented method of claim 1, wherein determining a first speech energy threshold based on the first orientation of the mobile device comprises selecting the first speech energy threshold from among a plurality of speech energy thresholds.

7. The computer-implemented method of claim 1, wherein detecting, using one or more processors, that the user of the mobile device has stopped speaking based on a comparison of first detected auditory information to the first speech energy threshold comprises:

comparing a volume of auditory input to the first speech energy threshold;

identifying that the volume of the auditory input is below the first speech energy threshold;

determining that the user of the mobile device has stopped speaking in response to identifying that the volume of the auditory input is below the first speech energy threshold.

8. The computer-implemented method of claim 1, wherein determining the first speech energy threshold based on the first orientation of the mobile device comprises:

determining a first operating mode of the mobile device based on the first orientation of the mobile device;

identifying first speech detection parameters that are based on the first operating mode of the mobile device, wherein the identified first speech detection parameters define the first speech energy threshold.

9. The computer-implemented method of claim 8, wherein determining the second speech energy threshold based on the second orientation of the mobile device comprises:

determining a second operating mode of the mobile device based on the second orientation of the mobile device, the second operating mode being distinct from the first operating mode;

identifying second speech detection parameters that are based on the second operating mode of the mobile device, wherein the identified second speech detection parameters define the second speech energy threshold.

10. The computer-implemented method of claim 1, wherein the first speech energy threshold is determined, at least in part, based on a distance between the mobile device and the user of the mobile device when the mobile device is in the first orientation.

11. A system, comprising:

one or more computers having;

at least one sensor that detects a first orientation and a second orientation of a mobile device;

a speech endpointer that identifies a first speech energy threshold based on the detected first orientation of the mobile device and a second speech energy threshold based on the detected second orientation of the mobile device;

the first speech energy threshold being usable to determine that a user of the mobile device has stopped speaking, the second speech energy threshold being usable to determine that the user of the mobile device has stopped speaking, and the first speech energy threshold being distinct from the second speech energy threshold.

12. The system of claim 11, wherein the at least one sensor comprises an accelerometer.

13. The system of claim 11, wherein the at least one sensor comprises a proximity sensor.

14. The system of claim 11, further comprising a gesture classifier that classifies movements of the mobile device.

15. A system, comprising:

one or more computers having;

at least one sensor that detects a first orientation and a second orientation of a mobile device;

means for identifying a first speech energy threshold based on the detected first orientation of the mobile device and a second speech energy threshold based on the detected second orientation of the mobile device,

the first speech energy threshold being usable to determine that a user of the mobile device has stopped speaking, the second speech energy threshold being usable to determine that the user of the mobile device has stopped speaking, and the first speech energy threshold being distinct from the second speech energy threshold.

16. The system of claim 15, wherein the at least one sensor comprises a proximity sensor.

17. The system of claim 15, further comprising a gesture classifier that classifies movements of the mobile device.

18. The system of claim 15, wherein the at least one sensor comprises a camera.

19. The system of claim 15, wherein the at least one sensor comprises an accelerometer.

20. A computer-implemented method, comprising:

determining a level of background noise in an operating environment of a mobile device;

comparing the determined level of background noise with a threshold level of background noise;

determining, using one or more processors and based on the comparison, a recommended pose for the mobile device that is associated with a relative orientation of the mobile device to a source of speech, wherein the determination occurs substantially before the speech is uttered by a user of the mobile device; and

outputting an indication of the recommended pose to the user of the mobile device, the indication identifying to the user a pose that the user should take with the mobile device.

21. The computer-implemented method of claim 20, wherein determining a level of background noise comprises determining a signal-to-noise ratio.

22. The computer-implemented method of claim 20, further comprising displaying a voice search option that enables input of one or more verbal search queries if the determined level of background noise is greater than the threshold level of background noise.

23. The computer-implemented method of claim 20, further comprising displaying a waveform representative of audio recorded by the mobile device.

24. The computer-implemented method of claim 20, further comprising displaying, as the user is speaking, a representation of audio being recorded by the mobile device.

25. The computer-implemented method of claim 24, wherein displaying, as the user is speaking, a representation of audio being recorded by the mobile device comprises displaying a representation of a signal-to-noise ratio.

26. The computer-implemented method of claim 25, wherein displaying, as the user is speaking, a representation of audio being recorded by the mobile device further comprises overlaying a representation of an amplitude of background noise recorded by the mobile device on the representation of the signal-to-noise ratio.

27. The computer-implemented method of claim 20, wherein determining the recommended pose comprises selecting the recommended pose from among a plurality of available poses in storage accessible by the mobile device.