Distributed endpointing for speech recognition

• US 9,818,407 B1
• Filed: 02/07/2013
• Issued: 11/14/2017
• Est. Priority Date: 02/07/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Secker-Walker, Hugh Evan, Basye, Kenneth John, Strom, Nikko, Thomas, Ryan Paul
• Primary Examiner(s): Zhu, Richard
• Application Number:
  US13/761,812
• Time in Patent Office: 1,741 Days
• Field of Search: 704248, 7042561-2568
• CPC Class Codes: G10L 15/04 Segmentation; Word boundary detection
• CPC Class Codes: G10L 15/142 Hidden Markov Models [HMMs]
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 25/18 the extracted parameters being spectral information of each sub-band
• CPC Class Codes: G10L 25/24 the extracted parameters being the cepstrum
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: G10L 25/87 Detection of discrete points within a voice signal

Abstract

An efficient audio streaming method and apparatus includes a client process implemented on a client or local device and a server process implemented on a remote server or server(s). The client process and server process each have speech recognition components and communicate over a network, and together efficiently manage the detection of speech in an audio signal streamed by the local device to the server for speech recognition and potentially further processing at the server. The client process monitors audio input and in a first detection stage, implements endpointing on the local device to determine when speech is detected. The client process may further determine if a “wakeword” is detected, and then the client process opens a connection and begins streaming audio to the server process via the network. The server process receives the speech audio stream and monitors the audio, implementing endpointing in the server process, to determine when to tell the client process to close the connection and stop streaming audio. The client process continues streaming audio to the server until the server process determines disconnect criteria have been met and tells the client process to stop streaming audio.

Claims

1. A system for performing distributed speech recognition, the system comprising:

a local device comprising at least one processor coupled to a memory, the memory including instructions operable to be executed by the processor to perform a set of actions, configuring the processor;

to receive audio using at least one microphone;

to monitor audio data corresponding to the audio to detect voice activity in the audio data,

to determine that the audio data comprises a wakeword,

to begin transmission of the audio data to a server device in response to determining the audio data comprises the wakeword,

to receive, from the server device, a confirmation that the audio data includes the wakeword,

to receive, from the server device, an indication to stop the transmission of the audio data;

to stop the transmission of the audio data in response to receiving the indication; and

to continue to receive further audio using the at least one microphone following receipt of the indication;

the server device comprising at least one processor coupled to a memory, the memory including instructions operable to be executed by the processor to perform a set of actions, configuring the processor;

to begin receiving the audio data,

to confirm the wakeword in the audio data,

to transmit the confirmation to the local device,

to determine an end of the voice activity in the audio data, and

to transmit the indication to the local device in response to determining the end of the voice activity.

2. The system of claim 1, wherein the processor of the local device is further configured to determine that the audio data includes the wakeword using Hidden Markov Model (HMM) techniques.

3. The system of claim 1, wherein the local device processor configured to monitor the audio data to detect the voice activity in the audio data comprises the local device processor further configured to detect the voice activity by evaluating quantitative aspects of the audio data selected from a group consisting of:

spectral slope between one or more frames of the audio data, energy levels of the audio data in one or more spectral bands, and signal-to-noise ratios of the audio data in one or more spectral bands.

4. A computer-implemented method, comprising:

receiving, by a local device, audio using at least one microphone;

monitoring, by the local device, audio data corresponding to the audio to detect voice activity in the audio data;

determining, by the local device, that the audio data comprises a wakeword;

starting, by the local device, transmission of the audio data to a remote device in response to determining the audio data comprises the wakeword;

receiving, by the local device, a confirmation from the remote device that the transmitted audio data includes the wakeword;

receiving, by the local device, an indication to stop the transmission of the audio data, from the remote device, in response to the remote device determining an end of the voice activity in the audio data;

stopping, by the local device, the transmission of the audio data in response to receiving the indication;

continuing to receive further audio using the at least one microphone following receipt of the indication.

5. The method of claim 4, wherein monitoring, by the local device, the audio data comprises determining a likelihood that the voice activity is present in the audio data by evaluating quantitative aspects of the audio data selected from a group consisting of:

spectral slope between one or more frames of the audio data, energy levels of the audio data in one or more spectral bands, and signal-to-noise ratios of the audio data in one or more spectral bands.

6. A computing device, comprising:

a processor;

a memory device including instructions operable to be executed by the processor to perform a set of actions, configuring the processor;

to receive audio using at least one microphone;

to monitor audio data corresponding to the audio to detect voice activity in the audio data;

to determine that the audio data comprises a wakeword;

to start transmission of the audio data to a remote device in response to determining the audio data comprises the wakeword;

to receive, from the remote device, a confirmation that the transmitted audio data includes the wakeword;

to receive an indication to stop the transmission of the audio data, from the remote device, in response to the remote device determining an end of the voice activity in the audio data;

to stop the transmission of the audio data in response to receiving the indication; and

to continue to receive further audio using the at least one microphone following receipt of the indication.

7. The computing device of claim 6, wherein the processor configured to monitor the audio data to detect the voice activity in the audio data comprises the processor configured to determine a likelihood that speech is present in the audio data.

8. The computing device of claim 6, wherein the processor is further configured to receive, from the remote device, speech recognition results based on the voice activity.

9. The computing device of claim 6, wherein the processor is further configured to stop the transmission upon expiration of a length of time.

10. The computing device of claim 6, wherein the processor is further configured to detect the voice activity in the audio data by evaluating quantitative aspects of the audio data selected from a group consisting of:

spectral slope between one or more frames of the audio data, energy levels of the audio data in one or more spectral bands, and signal-to-noise ratios of the audio data in one or more spectral bands.