Methods and systems for detecting and processing speech signals

• US 10,878,820 B2
• Filed: 02/20/2019
• Issued: 12/29/2020
• Est. Priority Date: 02/24/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Civelli, Jay Pierre, Shemer, Mikhal, Shabestary, Turaj Zakizadeh, Tapuska, David
• Primary Examiner(s): Chawan, Vijay B
• Application Number:
  US16/280,642
• Publication Number:
  US 20190189128A1
• Time in Patent Office: 678 Days
• Field of Search: 704270, 704275, 7042701, 704272, 704273, 704274, 704500
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

Provided are methods, systems, and apparatuses for detecting, processing, and responding to audio signals, including speech signals, within a designated area or space. A platform for multiple media devices connected via a network is configured to process speech, such as voice commands, detected at the media devices, and respond to the detected speech by causing the media devices to simultaneously perform one or more requested actions. The platform is capable of scoring the quality of a speech request, handling speech requests from multiple end points of the platform using a centralized processing approach, a de-centralized processing approach, or a combination thereof, and also manipulating partial processing of speech requests from multiple end points into a coherent whole when necessary.

Claims

1. A method comprising:

receiving, at a central processing device from each media device among multiple media devices located in a designated area and in communication with the central processing device, audio data captured by the media device and a corresponding audio quality measurement for the audio data, the audio data corresponding to a voice command spoken by a user in the designated area;

selecting, by the central processing device from among the multiple media devices, the media device that captured the audio data having a highest corresponding audio quality measurement to playout an audible response associated with the voice command; and

transmitting, by the central processing device, the audible response for the voice command to the selected media device, the audible response when received by the selected media device causing the selected media device to playout the audible response while the other media devices operate in a low power mode.

2. The method of claim 1, further comprising, after selecting the media device that captured the audio data having the highest corresponding audio quality measurement, transmitting, from the central processing device, an instruction to deactivate microphones on each of the other media devices.

3. The method of claim 1, wherein the corresponding audio quality measurement for the audio data captured by each media device among the multiple media devices is based on an estimated position of the user in relation to the media device.

4. The method of claim 3, wherein the estimated position of the user in relation to the media device is based on an angle of the user relative to the media device.

5. The method of claim 3, wherein the angle of the user relative to the media device is determined using a localizer of a beamformer.

6. The method of claim 1, wherein the corresponding audio quality measurement for the audio data captured by each media device among the multiple media devices is based on a corresponding signal power received at the media device when the audio data was captured.

7. The method of claim 1, wherein the corresponding audio quality measurement for the audio data captured by each media device among the multiple media devices is based on a corresponding hotword confidence score indicating a likelihood that the audio data captured by the media device includes a particular, predefined hotword.

8. The method of claim 7, wherein each media device among the multiple media devices provides the captured audio data to a hotword detector, the hotword detector configured to calculate the corresponding hotword confidence score indicating the likelihood that the audio data captured by the media device includes the particular, predefined hotword.

9. The method of claim 8, wherein the hotword detector utilizes a neural network to calculate the corresponding hotword confidence score.

10. The method of claim 1, wherein each media device among the multiple media devices operates in a low power mode and is configured to exit the low power mode upon detecting a particular hotword has likely been spoken.

11. A central processing device comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing device cause the data processing device to perform instructions comprising;

receiving, from each media device among multiple media devices located in a designated area and in communication with the central processing device, audio data captured by the media device and a corresponding audio quality measurement for the audio data, the audio data corresponding to a voice command spoken by a user in the designated area;

selecting, from among the multiple media devices, the media device that captured the audio data having a highest corresponding audio quality measurement to playout an audible response associated with the voice command; and

transmitting the audible response for the voice command to the selected media device, the audible response when received by the selected media device causing the selected media device to playout the audible response while the other media devices operate in a low power mode.

12. The central processing device of claim 11, wherein the operations further comprise, after selecting the media device that captured the audio data having the highest corresponding audio quality measurement, transmitting an instruction to deactivate microphones on each of the other media devices.

13. The central processing device of claim 11, wherein the corresponding audio quality measurement for the audio data captured by each media device among the multiple media devices is based on an estimated position of the user in relation to the media device.

14. The central processing device of claim 13, wherein the estimated position of the user in relation to the media device is based on an angle of the user relative to the media device.

15. The central processing device of claim 13, wherein the angle of the user relative to the media device is determined using a localizer of a beamformer.

16. The central processing device of claim 11, wherein the corresponding audio quality measurement for the audio data captured by each media device among the multiple media devices is based on a corresponding signal power received at the media device when the audio data was captured.

17. The central processing device of claim 11, wherein the corresponding audio quality measurement for the audio data captured by each media device among the multiple media devices is based on a corresponding hotword confidence score indicating a likelihood that the audio data captured by the media device includes a particular, predefined hotword.

18. The central processing device of claim 17, wherein each media device among the multiple media devices provides the captured audio data to a hotword detector, the hotword detector configured to calculate the corresponding hotword confidence score indicating the likelihood that the audio data captured by the media device includes the particular, predefined hotword.

19. The central processing device of claim 18, wherein the hotword detector utilizes a neural network to calculate the corresponding hotword confidence score.

20. The central processing device of claim 11, wherein each media device among the multiple media devices operates in a low power mode and is configured to exit the low power mode upon detecting a particular hotword has likely been spoken.