Method and apparatus for using image data to aid voice recognition

• US 9,747,900 B2
• Filed: 01/27/2014
• Issued: 08/29/2017
• Est. Priority Date: 05/24/2013
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Inventors: Zurek, Robert A, Schuster, Adrian M, Shau, Fu-Lin, Wu, Jincheng
• Primary Examiner(s): Riley, Marcus T
• Application Number:
  US14/164,354
• Publication Number:
  US 20140350924A1
• Time in Patent Office: 1,310 Days
• Field of Search: None
• CPC Class Codes: B60N 2/002 Seats provided with an occupancy detection means mounted therein or thereon detection means responsive to presence or absence of children B60N2/266
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 15/20 Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech G10L21/02 takes precedence
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04R 2430/20 Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic H04R2203/12 takes precedence
• CPC Class Codes: H04R 2460/07 Use of position data from wide-area or local-area positioning systems in hearing devices, e.g. program or information selection
• CPC Class Codes: H04R 2499/11 Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's

Abstract

A device performs a method for using image data to aid voice recognition. The method includes the device capturing image data of a vicinity of the device and adjusting, based on the image data, a set of parameters for voice recognition performed by the device. The set of parameters for the device performing voice recognition include, but are not limited to: a trigger threshold of a trigger for voice recognition; a set of beamforming parameters; a database for voice recognition; and/or an algorithm for voice recognition, wherein the algorithm can include using noise suppression or using acoustic beamforming.

Claims

1. A computer-implemented method comprising:

obtaining one or more images that are generated by one or more cameras of a mobile device;

analyzing one or more of the images;

identifying one or more features of an environment in which the mobile device is operating based on the analysis of one or more of the images;

determining, based on the identified one or more features, a trigger threshold against which respective values of voice commands are compared, each value of a voice command indicating a likelihood that received audio data corresponds to the voice command;

after determining the trigger threshold against which respective values voice commands are compared, receiving particular audio data;

determining that a value of a particular voice command associated with the particular audio data satisfies the trigger threshold; and

in response to determining that the value of the particular voice command associated with the particular audio data satisfies the trigger threshold, performing the particular voice command.

2. The method of claim 1, wherein the one or more features includes a number of persons in the environment in which the mobile device is operating.

3. The method of claim 1, wherein the one or more cameras comprise a front camera of the mobile device and a rear camera of the mobile device.

4. The method of claim 1, wherein determining a trigger threshold against which respective values measures of voice commands are compared comprises determining a phoneme matching threshold against which a number of matching phonemes between the voice commands and reference commands are compared.

5. The method of claim 1, wherein determining a trigger threshold against which respective values measures of voice commands are compared comprises determining a probability of the mobile device receiving audio from a source other than a speaker of the voice commands.

6. The method of claim 1, comprising:

determining that the mobile device is inside a motor vehicle; and

based on determining that the mobile device is inside the motor vehicle, reducing the trigger threshold.

7. The method of claim 1, comprising:

determining that one person is in the environment in which the mobile device is operating; and

based on determining that one person is in the environment in which the mobile device is operating, reducing the trigger threshold.

8. The method of claim 1, comprising:

determining that more than one person is in the environment in which the mobile device is operating; and

based on determining that more than one person is in the environment in which the mobile device is operating, increasing the trigger threshold.

23. The method of claim 1, comprising:

determining the confidence value associated with the particular voice command based on a degree to which phonemes of the particular voice command match phonemes stored as reference data.

9. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

obtaining one or more images that are generated by one or more cameras of a mobile device;

analyzing one or more of the images;

identifying one or more features of an environment in which the mobile device is operating based on the analysis of one or more of the images;

determining, based on the identified one or more features, a trigger threshold against which respective values of voice commands are compared, each value of a voice command indicating a likelihood that received audio data corresponds to the voice command;

after determining the trigger threshold against which respective values of voice commands are compared, receiving particular audio data;

determining that a value of a particular voice command associated with the particular audio data satisfies the trigger threshold; and

in response to determining that the value of the particular voice command associated with the particular audio data satisfies the trigger threshold, performing the particular voice command.

10. The system of claim 9, wherein the one or more features includes a number of persons in the environment in which the mobile device is operating.

11. The system of claim 9, wherein the one or more cameras comprise a front camera of the mobile device and a rear camera of the mobile device.

12. The system of claim 9, wherein determining a trigger threshold against which respective values measures of voice commands are compared comprises determining a phoneme matching threshold against which a number of matching phonemes between the voice commands and reference commands are compared.

13. The system of claim 9, wherein determining a trigger threshold against which respective values measures of voice commands are compared comprises determining a probability of the mobile device receiving audio from a source other than a speaker of the voice commands.

14. The system of claim 9, wherein the operations further comprise:

determining that the mobile device is inside a motor vehicle; and

based on determining that the mobile device is inside the motor vehicle, reducing the trigger threshold.

15. The system of claim 9, wherein the operations further comprise:

determining that one person is in the environment in which the mobile device is operating; and

based on determining that one person is in the environment in which the mobile device is operating, reducing the trigger threshold.

16. The system of claim 9, wherein the operations further comprise:

determining that more than one person is in the environment in which the mobile device is operating; and

based on determining that more than one person is in the environment in which the mobile device is operating, increasing the trigger threshold.

17. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

obtaining one or more images that are generated by one or more cameras of a mobile device;

analyzing one or more of the images;

identifying one or more features of an environment in which the mobile device is operating based on the analysis of one or more of the images;

determining, based on the identified one or more features, a trigger threshold against which respective values of voice commands are compared, each value of a voice command indicating a likelihood that received audio data corresponds to the voice command;

after determining the trigger threshold against which respective values of voice commands are compared, receiving particular audio data;

determining that a value of a particular voice command associated with the particular audio data satisfies the trigger threshold; and

in response to determining that the value of the particular voice command associated with the particular audio data satisfies the trigger threshold, performing the particular voice command.

18. The medium of claim 17, wherein the one or more features includes a number of persons in the environment in which the mobile device is operating.

19. The medium of claim 17, wherein the one or more cameras comprise a front camera of the mobile device and a rear camera of the mobile device.

20. The medium of claim 17, wherein determining a trigger threshold against which respective values of voice commands are compared comprises determining a phoneme matching threshold against which a number of matching phonemes between the voice commands and reference commands are compared.

21. The medium of claim 17, wherein the operations further comprise:

determining that the mobile device is inside a motor vehicle; and

based on determining that the mobile device is inside the motor vehicle, reducing the trigger threshold.

22. The medium of claim 17, wherein the operations further comprise:

determining that one person is in the environment in which the mobile device is operating; and

based on determining that one person is in the environment in which the mobile device is operating, reducing the trigger threshold.