Method and apparatus for using image data to aid voice recognition

• US 10,311,868 B2
• Filed: 03/21/2017
• Issued: 06/04/2019
• Est. Priority Date: 05/24/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Inventors: Zurek, Robert A., Schuster, Adrian M., Shau, Fu-Lin, Wu, Jincheng
• Primary Examiner(s): Riley, Marcus T
• Application Number:
  US15/464,704
• Publication Number:
  US 20170193996A1
• Time in Patent Office: 805 Days
• Field of Search: None
• CPC Class Codes: B60N 2/002 Seats provided with an occupancy detection means mounted therein or thereon detection means responsive to presence or absence of children B60N2/266
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 15/20 Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech G10L21/02 takes precedence
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04R 2430/20 Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic H04R2203/12 takes precedence
• CPC Class Codes: H04R 2460/07 Use of position data from wide-area or local-area positioning systems in hearing devices, e.g. program or information selection
• CPC Class Codes: H04R 2499/11 Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's

Abstract

A device performs a method for using image data to aid voice recognition. The method includes the device capturing (302) image data of a vicinity of the device and adjusting (304), based on the image data, a set of parameters for voice recognition performed by the device (102). The set of parameters for the device performing voice recognition include, but are not limited to: a trigger threshold of a trigger for voice recognition; a set of beamforming parameters; a database for voice recognition; and/or an algorithm for voice recognition. The algorithm may include using noise suppression or using acoustic beamforming.

Claims

1. A computer-implemented method comprising:

receiving, by a computing device, audio data corresponding to a user speaking in a vicinity of the computing device;

obtaining, by the computing device, a first image that includes a first representation of the user and that was captured during receipt of a first portion of the audio data;

obtaining, by the computing device, a second image that includes a second representation of the user and that was captured during receipt of a second portion of the audio data;

determining by the computing device, a first feature of the first representation of the user by analyzing the first image;

determining, by the computing device, a second feature of the second representation of the user by analyzing the second image;

based on the first feature of the first representation of the user included in the first image, obtaining, by the computing device, a transcription of the first portion of the audio data;

based on the second feature of the second representation of the user included in the second image, bypassing, by the computing device, obtaining a transcription of the second portion of the audio data; and

providing, for output by the computing device, the transcription of a portion of the audio data.

2. The method of claim 1, wherein the first feature of the first representation of the user included in the first image includes both eyes, a nose, and a mouth of the user.

3. The method of claim 1, wherein the second feature of the second representation of the user included in the second image includes less than both eyes, a nose, and a mouth of the user.

4. The method of claim 1, wherein:

the user is an authorized user of the computing device, and

providing the transcription of the portion of the audio data based on the user being an authorized user of the computing device.

5. The method of claim 1, wherein the first feature of the first representation of the user included in the first image includes both pupils of the user.

6. The method of claim 1, wherein the computing device obtains the first image and the second image using an infrared camera.

7. The method of claim 1, wherein an accuracy of speech recognition performed on the first portion and the second portion of the audio data varies according to a number of other users who are in each of the first image and the second image.

21. The method of claim 1, wherein:

obtaining a transcription of the first portion of the audio data comprises performing speech recognition on the first portion of the audio data, and

bypassing obtaining a transcription of the second portion of the audio data comprises bypassing performing speech recognition on the second portion of the audio data.

8. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving, by a computing device, audio data corresponding to a user speaking in a vicinity of the computing device;

obtaining, by the computing device, a first image that includes a first representation of the user and that was captured during receipt of a first portion of the audio data;

obtaining, by the computing device, a second image that includes a second representation of the user and that was captured during receipt of a second portion of the audio data;

determining, by the computing device, a first feature of the first representation of the user by analyzing the first image;

determining, by the computing device, a second feature of the second representation of the user by analyzing the second image;

based on the first feature of the first representation of the user included in the first image, obtaining, by the computing device, a transcription of the first portion of the audio data;

based on the second feature of the second representation of the user included in the second image, bypassing, by the computing device, obtaining a transcription of the second portion of the audio data; and

providing, for output by the computing device, the transcription of a portion of the audio data.

9. The system of claim 8, wherein the first feature of the first representation of the user included in the first image includes both eyes, a nose, and a mouth of the user.

10. The system of claim 8, wherein the second feature of the second representation of the user included in the second image includes less than both eyes, a nose, and a mouth of the user.

11. The system of claim 8, wherein:

the user is an authorized user of the computing device, and

providing the transcription of the portion of the audio data based on the user being an authorized user of the computing device.

12. The system of claim 8, wherein the first feature of the first representation of the user included in the first image includes both pupils of the user.

13. The system of claim 8, wherein the computing device obtains the first image and the second image using an infrared camera.

14. The system of claim 8, wherein an accuracy of speech recognition performed on the first portion and the second portion of the audio data varies according to a number of other users who are in each of the first image and the second image.

15. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

receiving, by a computing device, audio data corresponding to a user speaking in a vicinity of the computing device;

obtaining, by the computing device, a first image that includes a first representation of the user and that was captured during receipt of a first portion of the audio data;

obtaining, by the computing device, a second image that includes a second representation of the user and that was captured during receipt of a second portion of the audio data;

determining, by the computing device, a first feature of the first representation of the user by analyzing the first image;

determining, by the computing device, a second feature of the second representation of the user by analyzing the second image;

based on the first feature of the first representation of the user included in the first image, obtaining, by the computing device, a transcription of the first portion of the audio data;

based on the second feature of the second representation of the user included in the second image, bypassing, by the computing device, obtaining a transcription of the second portion of the audio data; and

providing, for output by the computing device, the transcription of a portion of the audio data.

16. The medium of claim 15, wherein the first feature of the first representation of the user included in the first image includes both eyes, a nose, and a mouth of the user.

17. The medium of claim 15, wherein the second feature of the second representation of the user included in the second image includes less than both eyes, a nose, and a mouth of the user.

18. The medium of claim 15, wherein:

the user is an authorized user of the computing device, and

providing the transcription of the portion of the audio data based on the user being an authorized user of the computing device.

19. The medium of claim 15, wherein the first feature of the first representation of the user included in the first image includes both pupils of the user.

20. The medium of claim 15, wherein the computing device obtains the first image and the second image using an infrared camera.