Augmenting Speech Segmentation and Recognition Using Head-Mounted Vibration and/or Motion Sensors

• US 20150356981A1
• Filed: 08/17/2015
• Published: 12/10/2015
• Est. Priority Date: 07/26/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Johnson, Michael Patrick, Dong, Jianchun, Balez, Mat
• Granted Patent: US 9,779,758 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G10L 15/20 Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech G10L21/02 takes precedence
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 21/16 Transforming into a non-visible representation devices or methods enabling ear patients to replace direct auditory perception by another kind of perception A61F11/04
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04R 1/46 Special adaptations for use as contact microphones, e.g. on musical instrument, on stethoscope throat mountings H04R1/14
• CPC Class Codes: H04R 2460/13 Hearing devices using bone conduction transducers

Abstract

Example methods and systems use multiple sensors to determine whether a speaker is speaking. Audio data in an audio-channel speech band detected by a microphone can be received. Vibration data in a vibration-channel speech band representative of vibrations detected by a sensor other than the microphone can be received. The microphone and the sensor can be associated with a head-mountable device (HMD). It is determined whether the audio data is causally related to the vibration data. If the audio data and the vibration data are causally related, an indication can be generated that the audio data contains HMD-wearer speech. Causally related audio and vibration data can be used to increase accuracy of text transcription of the HMD-wearer speech. If the audio data and the vibration data are not causally related, an indication can be generated that the audio data does not contain HMD-wearer speech.

Claims

1. A method, comprising:

receiving audio data representative of audio detected by a microphone, wherein the microphone is positioned on a head-mountable device (HMD), and wherein the received audio data comprises audio speech data in an audio-channel speech band;

receiving vibration data representative of vibrations detected by a sensor other than the microphone, wherein the sensor is positioned on the HMD, and wherein the received vibration data comprises vibration speech data in a vibration-channel speech band;

determining that the audio speech data-is causally related to the vibration speech data; and

in response to determining that the audio speech data is causally related to the vibration speech data, generating an indication that the audio data contains HMD-wearer speech.

2. The method of claim 1, further comprising:

determining a correlation delay between the microphone and the sensor.

3. The method of claim 1, further comprising:

in response to determining that the audio speech data is causally related to the vibration speech data, providing at least the audio speech data to a speech recognizer.

4. The method of claim 1, further comprising:

in response to determining that the audio speech data is causally related to the vibration speech data, providing the audio speech data and the vibration speech data to a speech recognizer.

5. The method of claim 4, further comprising:

recognizing text corresponding to the HMD-wearer speech in the audio speech data and the vibration speech data using the speech recognizer.

6. The method of claim 1, wherein the sensor comprises a bone-conducting microphone (BCM) and/or an accelerometer.

7. The method of claim 1, further comprising:

in response to determining that the audio speech data is causally related to the vibration speech data, conditioning the audio data and/or vibration data as speech.

21. The method of claim 1, further comprising:

determining a degree of spectral coherency between the audio speech data and the vibration speech data; and

wherein determining that the audio speech data is causally related to the vibration speech data comprises determining that the audio speech data is causally related to the vibration speech data based on the determined degree of spectral coherency.

8. A head-mountable device (HMD), comprising:

a processor;

a microphone;

a sensor;

a non-transitory computer-readable medium; and

program instructions stored on the non-transitory computer-readable medium, wherein the program instructions are executable by the processor to cause the HMD to perform functions comprising;

receiving audio data representative of audio detected by the microphone, wherein the received audio data comprises audio speech data in an audio-channel speech band;

receiving vibration data representative of vibrations detected by the sensor, wherein the received vibration data comprises vibration speech data in a vibration-channel speech band;

determining that the audio speech data-is causally related to the vibration speech data; and

in response to determining that the audio speech data-is causally related to the vibration speech data, generating an indication that the audio data contains HMD-wearer speech.

9. The HMD of claim 8, wherein the functions further comprise:

determining a correlation delay between the microphone and the sensor.

10. The HMD of claim 8, wherein the functions further comprise:

in response to determining that the audio speech data is causally related to the vibration speech data, providing at least the audio speech data to a speech recognizer.

11. The HMD of claim 8, wherein the functions further comprise:

in response to determining that the audio speech data is causally related to the vibration speech data, providing the audio speech data and the vibration speech data to a speech recognizer.

12. The HMD of claim 11, wherein the functions further comprise:

recognizing text corresponding to the HMD-wearer speech in the audio speech data and the vibration speech data using the speech recognizer.

13. The HMD of claim 8, wherein the sensor comprises a bone-conducting microphone (BCM) and/or an accelerometer.

14. The HMD of claim 8, wherein the functions further comprise:

in response to determining that the audio speech data is causally related to the vibration speech data, conditioning the audio data and/or vibration data as speech.

22. The HMD of claim 8, wherein the functions further comprise:

determining a degree of spectral coherency between the audio speech data and the vibration speech data; and

wherein determining that the audio speech data is causally related to the vibration speech data comprises determining that the audio speech data is causally related to the vibration speech data based on the determined degree of spectral coherency.

15. An article of manufacture including a non-transitory computer-readable medium having instructions stored thereon that, when executed by a computing device, cause the computing device to perform functions comprising:

receiving audio data representative of audio detected by a microphone positioned on a head-mountable display (HMD), wherein the received audio data comprises audio speech data in an audio-channel speech band;

receiving vibration data representative of vibrations detected by a sensor positioned on the HMD, wherein the received vibration data comprises vibration speech data in a vibration-channel speech band;

determining that the audio speech data is causally related to the vibration speech data; and

in response to determining that the audio speech data is causally related to the vibration speech data, generating an indication that the audio data contains HMD-wearer speech.

16. The article of manufacture of claim 15, wherein the functions further comprise:

determining a correlation delay between the microphone and the sensor.

17. The article of manufacture of claim 15, wherein the functions further comprise:

in response to determining that the audio speech data is causally related to the vibration speech data, providing the audio speech data and/or the vibration speech data to a speech recognizer.

18. The article of manufacture of claim 17, wherein the functions further comprise:

recognizing text corresponding to the HMD-wearer speech in the audio speech data and the vibration speech data using the speech recognizer.

19. The article of manufacture of claim 15, wherein the sensor comprises an accelerometer and/or a bone-conducting microphone (BCM).

20. The article of manufacture of claim 15, wherein the functions further comprise:

in response to determining that the audio speech data is causally related to the vibration speech data, conditioning the audio data and/or vibration data as speech.

23. The article of manufacture of claim 15, wherein the functions further comprise:

determining a degree of spectral coherency between the audio speech data and the vibration speech data; and

wherein determining that the audio speech data is causally related to the vibration speech data comprises determining that the audio speech data is causally related to the vibration speech data based on the determined degree of spectral coherency.