Pre-wakeword speech processing

• US 10,643,606 B2
• Filed: 01/24/2019
• Issued: 05/05/2020
• Est. Priority Date: 03/30/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Piersol, Kurt Wesley, Beddingfield, Gabriel
• Primary Examiner(s): Leland, III, Edwin S
• Application Number:
  US16/256,376
• Publication Number:
  US 20190156818A1
• Time in Patent Office: 467 Days
• Field of Search: 704254
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 25/87 Detection of discrete points within a voice signal

Abstract

A system for capturing and processing portions of a spoken utterance command that may occur before a wakeword. The system buffers incoming audio and indicates locations in the audio where the utterance changes, for example when a long pause is detected. When the system detects a wakeword within a particular utterance, the system determines the most recent utterance change location prior to the wakeword and sends the audio from that location to the end of the command utterance to a server for further speech processing.

Claims

1. A method comprising:

receiving audio;

storing, in non-transitory memory, audio data representing the audio;

determining a first location corresponding to a beginning of speech represented in the audio data;

determining a wake expression represented at a second location in the audio data, the second location being temporally subsequent to the first location in the audio data, wherein a representation of non-wake expression speech is located between the first location and the second location; and

after determining the wake expression, sending a portion of the audio data to a speech processing system, the portion of the audio data including the representation of non-wake expression speech.

2. The method of claim 1, further comprising:

storing an indicator representing the first location corresponds to a beginning of the speech; and

based at least in part on the indicator, determining the portion of the audio data is to include the representation of non-wake expression speech.

3. The method of claim 2, further comprising:

determining the indicator represents a closest preceding indicator to the wake expression in the audio data; and

based at least in part on determining the indicator represents the closest preceding indicator, determining the portion of the audio data is to include the representation of non-wake expression speech.

4. The method of claim 1, further comprising:

determining changes in frequencies of audio data at the second location in the audio data; and

determining the changes in frequencies correspond to a stored audio signature corresponding to the wake expression,

wherein determining the wake expression is represented at the second location is further based at least in part on determining the changes in frequencies correspond to the stored audio signature.

5. The method of claim 1, wherein determining the first location corresponds to the beginning of the speech comprises:

determining the first location corresponds to an amount of non-speech.

6. The method of claim 5, further comprising:

determining the amount of non-speech exceeds a threshold duration,

wherein determining the first location corresponds to the beginning of the speech is further based at least in part on determining the amount of non-speech exceeds the threshold duration.

7. The method of claim 6, further comprising:

determining a language of the speech; and

configuring the threshold duration based at least in part on the language.

8. The method of claim 6, further comprising:

determining an identifier of a speaker corresponding to the speech; and

configuring the threshold duration based at least in part on the identifier.

9. The method of claim 1, further comprising:

determining, in the audio data, a third location corresponding to an endpoint of the speech, the third location being temporally subsequent to the second location in the audio data; and

terminating, at the third location, the portion of the audio data sent to the speech processing system.

10. The method of claim 9, wherein determining the third location corresponds to the endpoint comprises:

determining the third location corresponds to an amount of non-speech.

11. A system comprising:

at least one processor;

at least one memory comprising instructions that, when executed by the at least one processor, cause the system to;

receive audio;

store, in non-transitory memory, audio data representing the audio;

determine a first location corresponding to a beginning of speech represented in the audio data;

determine a wake expression represented at a second location in the audio data, the second location being temporally subsequent to the first location in the audio data wherein a representation of non-wake expression speech is located between the first location and the second location; and

after determining the wake expression, send a portion of the audio data to a speech processing system, the portion of the audio data including the representation of non-wake expression speech.

12. The system of claim 11, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine changes in frequencies of audio data at the second location in the audio data; and

determine the changes in frequencies correspond to a stored audio signature corresponding to the wake expression,

wherein determining the wake expression is represented at the second location is further based at least in part on determining the changes in frequencies correspond to the stored audio signature.

13. The system of claim 11, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

store an indicator representing the first location corresponds to a beginning of the speech; and

based at least in part on the indicator, determine the portion of the audio data is to include the representation of non-wake expression speech.

14. The system of claim 13, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine the indicator represents a closest preceding indicator to the wake expression in the audio data; and

based at least in part on determining the indicator represents the closest preceding indicator, determine the portion of the audio data is to include the representation of non-wake expression speech.

15. The system of claim 14, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine the first location corresponds to an amount of non-speech,

wherein determining the first location corresponds to the beginning of the speech is further based at least in part on determining the first location corresponds to the amount of non-speech.

16. The system of claim 15, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine the amount of non-speech exceeds a threshold duration,

wherein determining the first location corresponds to the beginning of the speech is further based at least in part on determining the amount of non-speech exceeds the threshold duration.

17. The system of claim 16, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine a language of the speech; and

configure the threshold duration based at least in part on the language.

18. The system of claim 16, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine an identifier of a speaker corresponding to the speech; and

configure the threshold duration based at least in part on the identifier.

19. The system of claim 11, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine, in the audio data, a third location corresponding to an endpoint of the speech, the third location being temporally subsequent to the second location in the audio data; and

terminate, at the third location, the portion of the audio data sent to the speech processing system.

20. The system of claim 19, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine the third location corresponds to an amount of non-speech,

wherein determining the third location corresponds to the endpoint is further based at least in part on determining the third location corresponds to the amount of non-speech.