Multisensory speech detection

• US 10,026,419 B2
• Filed: 03/12/2015
• Issued: 07/17/2018
• Est. Priority Date: 11/10/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Burke, Dave, LeBeau, Michael J., Gianno, Konrad, Kristjansson, Trausti T., Jitkoff, John Nicholas, Senior, Andrew W.
• Primary Examiner(s): Toatley, Gregory J
• Assistant Examiner(s): STIFTER JR, TERENCE E
• Application Number:
  US14/645,802
• Publication Number:
  US 20150287423A1
• Time in Patent Office: 1,223 Days
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/10 using distance or distortion measures between unknown speech and reference templates
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 25/21 the extracted parameters being power information
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04M 2250/74 with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00
• CPC Class Codes: H04R 1/08 Mouthpieces; Microphones; Attachments therefor
• CPC Class Codes: H04W 4/026 using orientation information, e.g. compass

Abstract

A computer-implemented method of multisensory speech detection is disclosed. The method comprises determining an orientation of a mobile device and determining an operating mode of the mobile device based on the orientation of the mobile device. The method further includes identifying speech detection parameters that specify when speech detection begins or ends based on the determined operating mode and detecting speech from a user of the mobile device based on the speech detection parameters.

Claims

1. A computer-implemented method, comprising:

initiating, by a mobile computing device, an audio recording process using a microphone of the mobile computing device;

identifying, by the mobile computing device, a first orientation of the mobile computing device at a beginning of the audio recording process;

determining, during the audio recording process, that the mobile computing device has transitioned from the first orientation to a second orientation;

in response to determining that the mobile computing device has transitioned from the first orientation to a second orientation, determining a speech energy threshold for comparing to speech received during the audio recording process and while the mobile computing device is in the second orientation;

comparing an energy of the speech received during the audio recording process and while the mobile computing device is in the second orientation to the speech energy threshold;

determining an end of speech condition based on (i) the energy of the speech received during the audio recording process and while the mobile computing device is in the second orientation and (ii) the speech energy threshold; and

stopping, based on determining the end of speech condition, the audio recording process.

2. The method of claim 1, wherein identifying the first orientation of the mobile computing device is performed in response to a proximity sensor of the mobile computing device detecting an object within a predetermined distance of the proximity sensor.

3. The method of claim 1, wherein identifying the first orientation of the mobile computing device comprises detecting that a user is within a predetermined distance of a proximity sensor of the mobile computing device, and wherein determining that the mobile computing device has transitioned from the first orientation to the second orientation includes determining that the user is no longer within the predetermined distance of the proximity sensor.

4. The method of claim 1, wherein determining that the mobile computing device has transitioned from the first orientation to the second orientation is performed using one or more accelerometers of the mobile computing device.

5. The method of claim 1, comprising:

classifying the transition of the mobile computing device from the first orientation to the second orientation as one of a “

to mouth”

gesture, a “

from mouth”

gesture, a “

to ear”

gesture, or a “

from ear”

gesture,

wherein determining the speech energy threshold for comparing to speech received during the audio recording process and while the mobile computing device is in the second orientation is based on classifying the transition of the mobile computing device from the first orientation to the second orientation as one of the “

to mouth”

gesture, the “

from mouth”

gesture, the “

to ear”

gesture, or the “

from ear”

gesture.

6. The method of claim 1, wherein determining, during the audio recording process, that the mobile computing device has transitioned from the first orientation to the second orientation comprises determining that an angle between a user and the mobile computing device is outside of a selected set of angles.

7. The method of claim 1, wherein determining, during the audio recording process, that the mobile computing device has transitioned from the first orientation to the second orientation comprises determining that the mobile computing device is beyond a predetermined distance from a portion of a user.

8. The method of claim 1, comprising:

while the mobile computing device is in the first orientation, comparing the energy of the speech received during the audio recording process to a different speech energy threshold;

determining that the energy of the speech received during the audio recording process and while the mobile computing device is in the first orientation does is greater than the different speech energy threshold; and

based on determining that the energy of the speech received during the audio recording process and while the mobile computing device is in the first orientation is greater than the different speech energy threshold, continuing the audio recording process.

9. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

initiating, by a mobile computing device, an audio recording process using a microphone of the mobile computing device;

identifying, by the mobile computing device, a first orientation of the mobile computing device at a beginning of the audio recording process;

determining, during the audio recording process, that the mobile computing device has transitioned from the first orientation to a second orientation;

in response to determining that the mobile computing device has transitioned from the first orientation to a second orientation, determining a speech energy threshold for comparing to speech received during the audio recording process and while the mobile computing device is in the second orientation;

comparing an energy of the speech received during the audio recording process and while the mobile computing device is in the second orientation to the speech energy threshold;

determining an end of speech condition based on (i) the energy of the speech received during the audio recording process and while the mobile computing device is in the second orientation and (ii) the speech energy threshold; and

stopping, based on determining the end of speech condition, the audio recording process.

10. The computer storage medium of claim 9, wherein identifying the first orientation of the mobile computing device is performed in response to a proximity sensor of the mobile computing device detecting an object within a predetermined distance of the proximity sensor.

11. The computer storage medium of claim 9, wherein identifying the first orientation of the mobile computing device comprises detecting that a user is within a predetermined distance of a proximity sensor of the mobile computing device, and wherein determining that the mobile computing device has transitioned from the first orientation to the second orientation includes determining that the user is no longer within the predetermined distance of the proximity sensor.

12. The computer storage medium of claim 9, wherein determining that the mobile computing device has transitioned from the first orientation to the second orientation is performed using one or more accelerometers of the mobile computing device.

13. The computer storage medium of claim 9, comprising:

classifying the transition of the mobile computing device from the first orientation to the second orientation as one of a “

to mouth”

gesture, a “

from mouth”

gesture, a “

to ear”

gesture, or a “

from ear”

gesture,

wherein determining the speech energy threshold for comparing to speech received during the audio recording process and while the mobile computing device is in the second orientation is based on classifying the transition of the mobile computing device from the first orientation to the second orientation as one of the “

to mouth”

gesture, the “

from mouth”

gesture, the “

to ear”

gesture, or the “

from ear”

gesture.

14. The computer storage medium of claim 9, wherein determining, during the audio recording process, that the mobile computing device has transitioned from the first orientation to the second orientation comprises determining that an angle between a user and the mobile computing device is outside of a selected set of angles.

15. The computer storage medium of claim 9, wherein determining, during the audio recording process, that the mobile computing device has transitioned from the first orientation to the second orientation comprises determining that the mobile computing device is beyond a predetermined distance from a portion of a user.

16. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

initiating, by a mobile computing device, an audio recording process using a microphone of the mobile computing device;

identifying, by the mobile computing device, a first orientation of the mobile computing device at a beginning of the audio recording process;

determining, during the audio recording process, that the mobile computing device has transitioned from the first orientation to a second orientation;

in response to determining that the mobile computing device has transitioned from the first orientation to a second orientation, determining a speech energy threshold for comparing to speech received during the audio recording process and while the mobile computing device is in the second orientation;

comparing an energy of the speech received during the audio recording process and while the mobile computing device is in the second orientation to the speech energy threshold;

determining an end of speech condition based on (i) the energy of the speech received during the audio recording process and while the mobile computing device is in the second orientation and (ii) the speech energy threshold; and

stopping, based on determining the end of speech condition, the audio recording process.

17. The system of claim 16, wherein identifying the first orientation of the mobile computing device is performed in response to a proximity sensor of the mobile computing device detecting an object within a predetermined distance of the proximity sensor.

18. The system of claim 16, wherein identifying the first orientation of the mobile computing device comprises detecting that a user is within a predetermined distance of a proximity sensor of the mobile computing device, and wherein determining that the mobile computing device has transitioned from the first orientation to the second orientation includes determining that the user is no longer within the predetermined distance of the proximity sensor.

19. The system of claim 16, wherein determining that the mobile computing device has transitioned from the first orientation to the second orientation is performed using one or more accelerometers of the mobile computing device.

20. The system of claim 16, wherein the operations further comprise:

classifying the transition of the mobile computing device from the first orientation to the second orientation as one of a “

to mouth”

gesture, a “

from mouth”

gesture, a “

to ear”

gesture, or a “

from ear”

gesture,

wherein determining the speech energy threshold for comparing to speech received during the audio recording process and while the mobile computing device is in the second orientation is based on classifying the transition of the mobile computing device from the first orientation to the second orientation as one of the “

to mouth”

gesture, the “

from mouth”

gesture, the “

to ear”

gesture, or the “

from ear”

gesture.