COLLABORATIVE VOICE CONTROLLED DEVICES

• US 20180182397A1
• Filed: 12/22/2016
• Published: 06/28/2018
• Est. Priority Date: 12/22/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Carbune, Victor, Gonnet Anders, Pedro, Deselaers, Thomas, Feuz, Sandro
• Granted Patent: US 10,559,309 B2
• Time in Patent Office: Days
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for collaboration between multiple voice controlled devices are disclosed. In one aspect, a method includes the actions of identifying, by a first computing device, a second computing device that is configured to respond to a particular, predefined hotword; receiving audio data that corresponds to an utterance; receiving a transcription of additional audio data outputted by the second computing device in response to the utterance; based on the transcription of the additional audio data and based on the utterance, generating a transcription that corresponds to a response to the additional audio data; and providing, for output, the transcription that corresponds to the response.

Claims

1. A computer-implemented method, comprising:

identifying, by a first computing device, a second computing device that is configured to respond to a particular, predefined hotword;

receiving audio data that corresponds to an utterance;

receiving a transcription of additional audio data outputted by the second computing device in response to the utterance;

based on the transcription of the additional audio data and based on the utterance, generating a transcription that corresponds to a response to the additional audio data; and

providing, for output, the transcription that corresponds to the response.

2. The method of claim 1, wherein providing, for output the transcription that corresponds to the response comprises:

providing an initial portion of the transcription to a speech synthesizer of the first computing device; and

providing, to the second computing device, (i) a remaining portion of the transcription and (ii) instructions to output the remaining portion of the transcription using a speech synthesizer of the second computing device.

3. The method of claim 1, comprising:

before performing speech recognition on the audio data that corresponds to the utterance;

determining that the audio data that corresponds to the utterance includes the particular, predefined hotword; and

receiving data indicating that the second computing device is responding to the audio data.

4. The method of claim 1, wherein providing, for output, the transcription that corresponds to the response comprises:

providing the transcription that corresponds to the response to a speech synthesizer.

5. The method of claim 3, wherein audio data that corresponds to the synthesized transcription is received by a third computing device that is configured to generate a response based on the audio data that corresponds to the synthesized transcription, the transcription of the additional audio data, and the utterance

6. The method of claim 1, comprising:

after providing, for output, the transcription that corresponds to the response, receiving, by the first computing device, audio data that corresponds to a second utterance;

based on the transcription of the additional audio data, based on the utterance, and based on the second utterance, generating an additional transcription that corresponds to a response to the audio data that corresponds to the second utterance; and

providing, for output, the additional transcription.

7. The method of claim 3, comprising:

receiving data indicating a type of speech synthesizer used by the second computing device to output the additional audio data,

wherein a type of speech synthesizer that received the transcription that corresponds to the response is different than the type of speech synthesizer used by the second computing device to output the additional audio data.

8. The method of claim 1, wherein providing, for output, the transcription that corresponds to the response comprises:

providing the transcription to a display of the first computing device.

9. The method of claim 2, wherein receiving data indicating that the second computing device is responding to the audio data comprises:

receiving, from the second computing device, a short range radio signal that indicates that the second computing device is responding to the audio data;

receiving, from the second computing device and through a local network, data indicating that the second computing device is responding to the audio data;

or

receiving, from a server, data indicating that the second computing device is responding to the audio data.

10. The method of claim 2, wherein determining that the audio data that corresponds to the utterance includes the particular, predefined hotword comprises:

extracting audio features of the audio data that corresponds to the utterance;

generating a hotword confidence score by processing the audio features;

determining that the hotword confidence score satisfies a hotword confidence threshold; and

based on determining that the hotword confidence score satisfies a hotword confidence threshold, determining that the audio data that corresponds to the utterance includes the particular, predefined hotword.

11. The method of claim 1, wherein generating a transcription that corresponds to a response to the additional audio data comprises:

determining user information that is associated with a first user of the first computing device or with a second user of the second computing device; and

wherein the transcription is generated based on the user information.

12. The method of claim 1, wherein generating a transcription that corresponds to a response to the additional audio data comprises:

accessing data that is associated with the transcription of the additional audio data; and

wherein the transcription is generated based on the accessed data.

13. The method of claim 1, comprising:

determining a location of the first computing device,

wherein the transcription is generated based on the location of the first computing device.

14. The method of claim 2, comprising:

in response to receiving data indicating that the second computing device is responding to the audio data, providing, to the second computing device or to a server, the audio data that corresponds to the utterance.

15. The method of claim 1, comprising:

generating a first audio fingerprint of the audio data that corresponds to the utterance;

receiving, from the second computing device, a second audio fingerprint of the audio data that corresponds to the utterance;

comparing the first audio fingerprint to the second audio fingerprint; and

based on comparing the first audio fingerprint to the second audio fingerprint, determining that the audio data received by the first computing device corresponds to the audio data received by the second computing device.

16. The method of claim 1, comprising:

receiving, from the second computing device or a server, a transcription of the audio data that corresponds to the utterance,

wherein generating the transcription that corresponds to the response to the additional audio data is based further on the transcription of the audio data that corresponds to the utterance.

17. The method of claim 1, wherein the first computing device and the second computing device are able to detect short range radio communications from each other.

18. The method of claim 1, wherein the first computing device and the second computing device are co-located.

19. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

identifying, by a first computing device, a second computing device that is configured to respond to a particular, predefined hotword;

receiving audio data that corresponds to an utterance;

receiving a transcription of additional audio data outputted by the second computing device in response to the utterance;

based on the transcription of the additional audio data and based on the utterance, generating a transcription that corresponds to a response to the additional audio data; and

providing, for output, the transcription that corresponds to the response.

20. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

identifying, by a first computing device, a second computing device that is configured to respond to a particular, predefined hotword;

receiving audio data that corresponds to an utterance;

receiving a transcription of additional audio data outputted by the second computing device in response to the utterance;

based on the transcription of the additional audio data and based on the utterance, generating a transcription that corresponds to a response to the additional audio data; and

providing, for output, the transcription that corresponds to the response.