Hotword detection on multiple devices

• US 9,990,922 B2
• Filed: 06/23/2016
• Issued: 06/05/2018
• Est. Priority Date: 10/09/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Foerster, Jakob Nicolaus, Gruenstein, Alexander H.
• Primary Examiner(s): Hang, Vu B
• Application Number:
  US15/190,739
• Publication Number:
  US 20160300571A1
• Time in Patent Office: 712 Days
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2025/783 based on threshold decision
• CPC Class Codes: G10L 25/03 characterised by the type of extracted parameters
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: G10L 25/87 Detection of discrete points within a voice signal

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for hotword detection on multiple devices are disclosed. In one aspect, a method includes the actions of receiving, by a computing device, audio data that corresponds to an utterance. The actions further include determining a likelihood that the utterance includes a hotword. The actions further include determining a loudness score for the audio data. The actions further include based on the loudness score, determining an amount of delay time. The actions further include, after the amount of delay time has elapsed, transmitting a signal that indicates that the computing device will initiate speech recognition processing on the audio data.

Claims

1. A computer-implemented method comprising:

receiving, by a mobile computing device that is (i) configured to process voice commands that are preceded by a predefined hotword, and (ii) is in proximity to another mobile computing device that is also configured to process voice commands that are preceded by the same, predefined hotword, an audio input representing an utterance by the speaker of a voice command that is preceded by the predefined hotword;

while receiving the audio input representing the utterance by the speaker of the voice command that is preceded by the predefined hotword, performing, by the mobile computing device, an operation;

after receiving the audio input representing the utterance by the speaker of the voice command that is preceded by the predefined hotword, receiving an ultrasonic signal from the other mobile computing device;

in response to receiving the ultrasonic signal from the other mobile computing device, (i) placing the mobile device into a sleep mode, (ii) bypassing, by the mobile computing device, further processing of the voice command, (iii) bypassing, by the mobile computing device, emitting an ultrasonic signal, and (iv) bypassing, by the mobile computing device, outputting a visual indication that the mobile computing device is processing the voice command; and

while receiving the ultrasonic signal from the other mobile computing device and bypassing processing of the voice command, continuing, by the mobile computing device, to perform the operation without interruption.

2. The method of claim 1, comprising:

determining a variable delay period based on the audio input representing the utterance by the speaker of the voice command that is preceded by the predefined hotword,

wherein a period between receiving the audio input representing the utterance and receiving the ultrasonic signal from the other mobile computing device is less than the variable delay period.

3. The method of claim 1, wherein:

while receiving the audio input representing the utterance by the speaker of the voice command that is preceded by the predefined hotword, a screen of the mobile computing device is blank, and

while receiving the ultrasonic signal from the other mobile computing device, and bypassing processing of the voice command, the screen of the mobile computing device is blank.

10. The method of claim 1, comprising:

determining a loudness of the audio input representing the utterance based on a sound pressure level of the audio input,

wherein (i) placing the mobile device into the sleep mode, (ii) bypassing, by the mobile computing device, further processing of the voice command, (iii) bypassing, by the mobile computing device, emitting the ultrasonic signal, and (iv) bypassing, by the mobile computing device, outputting the visual indication that the mobile computing device is processing the voice command is based on the loudness of the audio input representing the utterance.

11. The method of claim 1, wherein the operation comprises playing a video.

15. The method of claim 1, comprising, in response to receiving the ultrasonic signal from the other mobile computing device, (i) determining, by the mobile computing device, to place the mobile device into the sleep mode, (ii) determining, by the mobile computing device, to bypass further processing of the voice command, (iii) determining, by the mobile computing device, to bypass emitting the ultrasonic signal, and (iv) determining, by the mobile computing device, to bypass outputting the visual indication that the mobile computing device is processing the voice command.

4. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving, by a mobile computing device that is (i) configured to process voice commands that are preceded by a predefined hotword, and (ii) is in proximity to another mobile computing device that is also configured to process voice commands that are preceded by the same, predefined hotword, an audio input representing an utterance by the speaker of a voice command that is preceded by the predefined hotword;

while receiving the audio input representing the utterance by the speaker of the voice command that is preceded by the predefined hotword, performing, by the mobile computing device, an operation;

after receiving the audio input representing the utterance by the speaker of the voice command that is preceded by the predefined hotword, receiving an ultrasonic signal from the other mobile computing device;

in response to receiving the ultrasonic signal from the other mobile computing device, (i) placing the mobile device into a sleep mode, (ii) bypassing, by the mobile computing device, further processing of the voice command, (iii) bypassing, by the mobile computing device, emitting an ultrasonic signal, and (iv) bypassing, by the mobile computing device, outputting a visual indication that the mobile computing device is processing the voice command; and

while receiving the ultrasonic signal from the other mobile computing device and bypassing processing of the voice command, continuing, by the mobile computing device, to perform the operation without interruption.

5. The system of claim 4, wherein the operations further comprise:

determining a variable delay period in response to receiving the audio input representing the utterance by the speaker of the voice command that is preceded by the predefined hotword,

wherein a period between receiving the audio input representing the utterance and receiving the ultrasonic signal from the other mobile computing device is less than the variable delay period.

6. The system of claim 4, wherein:

while receiving the audio input representing the utterance by the speaker of the voice command that is preceded by the predefined hotword, a screen of the mobile computing device is blank, and

while receiving the signal from the other mobile computing device, and bypassing processing of the voice command, the screen of the mobile computing device is blank.

12. The system of claim 4, comprising:

determining a loudness of the audio input representing the utterance based on a sound pressure level of the audio input,

wherein (i) placing the mobile device into the sleep mode, (ii) bypassing, by the mobile computing device, further processing of the voice command, (iii) bypassing, by the mobile computing device, emitting the ultrasonic signal, and (iv) bypassing, by the mobile computing device, outputting the visual indication that the mobile computing device is processing the voice command is based on the loudness of the audio input representing the utterance.

13. The system of claim 4, wherein the operation comprises playing a video.

7. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

receiving, by a mobile computing device that is (i) configured to process voice commands that are preceded by a predefined hotword, and (ii) is in proximity to another mobile computing device that is also configured to process voice commands that are preceded by the same, predefined hotword, an audio input representing an utterance by the speaker of a voice command that is preceded by the predefined hotword;

while receiving the audio input representing the utterance by the speaker of the voice command that is preceded by the predefined hotword, performing, by the mobile computing device, an operation;

after receiving the audio input representing the utterance by the speaker of the voice command that is preceded by the predefined hotword, receiving an ultrasonic signal from the other mobile computing device;

in response to receiving the ultrasonic signal from the other mobile computing device, (i) placing the mobile device into a sleep mode, (ii) bypassing, by the mobile computing device, further processing of the voice command, (iii) bypassing, by the mobile computing device, emitting an ultrasonic signal, and (iv) bypassing, by the mobile computing device, outputting a visual indication that the mobile computing device is processing the voice command; and

while receiving the ultrasonic signal from the other mobile computing device and bypassing processing of the voice command, continuing, by the mobile computing device, to perform the operation without interruption.

8. The medium of claim 7, wherein the operations further comprise:

determining a variable delay period in response to receiving the audio input representing the utterance by the speaker of the voice command that is preceded by the predefined hotword,

wherein a period between receiving the audio input representing the utterance and receiving the ultrasonic signal from the other mobile computing device is less than the variable delay period.

9. The medium of claim 7, wherein:

while receiving the audio input representing the utterance by the speaker of the voice command that is preceded by the predefined hotword, a screen of the mobile computing device is blank, and

while receiving the signal from the other mobile computing device, and bypassing processing of the voice command, the screen of the mobile computing device is blank.

14. The medium of claim 7, wherein the operation comprises playing a video.