Multisensory speech detection

• US 9,570,094 B2
• Filed: 06/29/2015
• Issued: 02/14/2017
• Est. Priority Date: 11/10/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Burke, Dave, LeBeau, Michael J., Gianno, Konrad, Kristjansson, Trausti T., Jitkoff, John Nicholas, Senior, Andrew W.
• Primary Examiner(s): Chawan, Vijay B
• Application Number:
  US14/753,904
• Publication Number:
  US 20150302870A1
• Time in Patent Office: 596 Days
• Field of Search: 704/246, 704/270, 704/275, 704/2, 704/9, 704/270.1, 345/156, 345/2.2, 345/419, 345/633, 455/414.1, 455/563, 455/567, 455/566, 348/51, 348/734, 348/745, 379/45
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/10 using distance or distortion measures between unknown speech and reference templates
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 25/21 the extracted parameters being power information
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04M 2250/74 with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00
• CPC Class Codes: H04R 1/08 Mouthpieces; Microphones; Attachments therefor
• CPC Class Codes: H04W 4/026 using orientation information, e.g. compass

Abstract

A computer-implemented method of multisensory speech detection is disclosed. The method comprises determining an orientation of a mobile device and determining an operating mode of the mobile device based on the orientation of the mobile device. The method further includes identifying speech detection parameters that specify when speech detection begins or ends based on the determined operating mode and detecting speech from a user of the mobile device based on the speech detection parameters.

Claims

1. A computer-implemented method, comprising:

identifying, by a mobile computing device, an identity of a person that is speaking into the mobile computing device;

selecting stored biometric information associated with the identified person;

determining, by the mobile computing device, that the identified person has stopped speaking into the mobile computing device based at least on the selected biometric information associated with the identified person; and

generating a transcription of the person'"'"'s speech into the mobile device in response to determining that, based at least on the selected biometric information associated with the identified person, the identified person has stopped speaking into the mobile computing device.

2. The method of claim 1, wherein identifying the identity of the person that is speaking into the mobile computing device comprises comparing auditory information received by a microphone of the mobile computing device to stored auditory profile information for the person that is speaking into the mobile computing device.

3. The method of claim 2, wherein identifying the identity of the person that is speaking into the mobile computing device further comprises comparing the auditory information received by the microphone of the mobile computing device to stored auditory profile information for one or more persons other than the person speaking that is into the mobile computing device.

4. The method of claim 1, wherein identifying the identity of the person that is speaking into the mobile computing device comprises using additional biometric information obtained by a camera of the mobile computing device to identify the identity of the person that is speaking into the mobile computing device.

5. The method of claim 1, wherein determining that the identified person has stopped speaking into the mobile computing device is based further on information that is specific to the identified person.

6. The method of claim 1, wherein identifying an identity of a person that is speaking into the mobile computing device determining that the person'"'"'s speech is not background noise.

7. The method of claim 6, wherein determining that the person'"'"'s speech is not background noise includes comparing auditory information received by a microphone of the mobile computing device to a speech energy threshold for the person.

8. The method of claim 1, wherein determining that the identified person has stopped speaking into the mobile computing device is based further on an input provided by a pose identifier, an input provided by a speech detector, and an input provided by a speaker identifier.

17. The method of claim 1, wherein the stored biometric information includes face appearance, ear shape, or hand print.

19. The system of claim 17, wherein identifying the identity of the person that is speaking into the mobile computing device comprises comparing auditory information received by a microphone of the mobile computing device to stored auditory profile information for the person that is speaking into the mobile computing device.

20. The system of claim 17, wherein determining that the identified person has stopped speaking into the mobile computing device is based further on an input provided by a pose identifier, an input provided by a speech detector, and an input provided by a speaker identifier.

9. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

identifying, by a mobile computing device, an identity of a person that is speaking into the mobile computing device;

selecting stored biometric information associated with the identified person;

determining, by the mobile computing device, that the identified person has stopped speaking into the mobile computing device based at least on the selected biometric information associated with the identified person; and

generating a transcription of the person'"'"'s speech into the mobile device in response to determining that, based at least on the selected biometric information associated with the identified person, the identified person has stopped speaking into the mobile computing device.

10. The computer storage medium of claim 9, wherein identifying the identity of the person that is speaking into the mobile computing device comprises comparing auditory information received by a microphone of the mobile computing device to stored auditory profile information for the person that is speaking into the mobile computing device.

11. The computer storage medium of claim 10, wherein identifying the identity of the person that is speaking into the mobile computing device further comprises comparing the auditory information received by the microphone of the mobile computing device to stored auditory profile information for one or more persons other than the person speaking that is into the mobile computing device.

12. The computer storage medium of claim 9, wherein identifying the identity of the person that is speaking into the mobile computing device comprises using additional biometric information obtained by a camera of the mobile computing device to identify the identity of the person that is speaking into the mobile computing device.

13. The computer storage medium of claim 9, wherein determining that the identified person has stopped speaking into the mobile computing device is based further on information that is specific to the identified person.

14. The computer storage medium of claim 9, wherein identifying an identity of a person that is speaking into the mobile computing device determining that the person'"'"'s speech is not background noise.

15. The computer storage medium of claim 14, wherein determining that the person'"'"'s speech is not background noise includes comparing auditory information received by a microphone of the mobile computing device to a speech energy threshold for the person.

16. The computer storage medium of claim 9, wherein determining that the identified person has stopped speaking into the mobile computing device is based further on an input provided by a pose identifier, an input provided by a speech detector, and an input provided by a speaker identifier.

18. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

identifying, by a mobile computing device, an identity of a person that is speaking into the mobile computing device;

selecting stored biometric information associated with the identified person;

determining, by the mobile computing device, that the identified person has stopped speaking into the mobile computing device based at least on the selected biometric information associated with the identified person; and

generating a transcription of the person'"'"'s speech into the mobile device in response to determining that, based at least on the selected biometric information associated with the identified person, the identified person has stopped speaking into the mobile computing device.