Method and apparatus for evaluating trigger phrase enrollment

• US 10,163,438 B2
• Filed: 05/25/2017
• Issued: 12/25/2018
• Est. Priority Date: 07/31/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Inventors: Clark, Joel A., Ramabadran, Tenkasi V., Jasiuk, Mark A.
• Primary Examiner(s): Han, Qi
• Application Number:
  US15/605,565
• Publication Number:
  US 20170263243A1
• Time in Patent Office: 579 Days
• Field of Search: 704244, 704231, 704233, 704235, 704246, 704249, 704250, 704251, 704255
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/1807 using prosody or stress
• CPC Class Codes: G10L 15/20 Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech G10L21/02 takes precedence
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 21/0264 characterised by the type of parameter measurement, e.g. correlation techniques, zero crossing techniques or predictive techniques
• CPC Class Codes: G10L 25/84 for discriminating voice from noise

Abstract

An electronic device includes a microphone that receives an audio signal that includes a spoken trigger phrase, and a processor that is electrically coupled to the microphone. The processor measures characteristics of the audio signal, and determines, based on the measured characteristics, whether the spoken trigger phrase is acceptable for trigger phrase model training. If the spoken trigger phrase is determined not to be acceptable for trigger phrase model training, the processor rejects the trigger phrase for trigger phrase model training.

Claims

1. A computer-implemented method comprising:

during a trigger phrase enrollment process;

prompting, by a speech recognition-enabled electronic device, a user of the speech recognition-enabled electronic device to speak a trigger phrase;

receiving, at the speech recognition-enabled device, a first audio signal corresponding to the user speaking the trigger phrase;

determining, by the speech recognition-enabled device, a number of segments in the first audio signal that have voice activity, each segment comprising a corresponding sequence of continuous frames in the first audio signal that have voice activity; and

when the number of segments in the first audio signal that have voice activity satisfies a threshold value, training, by the speech recognition-enabled electronic device, a trigger phrase model with the first audio signal corresponding to the user speaking the trigger phrase, the trigger phrase model configured to detect the trigger phrase in a spoken utterance; and

after the trigger phrase enrollment process;

receiving, at the speech recognition-enabled device and while the speech recognition-enabled electronic device is in a sleep mode, a second audio signal including an utterance of the trigger phrase spoken by the user; and

detecting, by the speech recognition-enabled electronic device and using the trigger phrase model trained during the trigger phrase enrollment process, the utterance of the trigger phrase in the second audio signal, the trigger phrase when detected in the second audio signal causing the speech recognition-enabled electronic device to wake from the sleep mode, the sleep mode comprising a power-saving mode of operation in which one or more parts of the speech recognition-enabled electronic device are in a low-power state or powered off.

2. The computer-implemented method of claim 1, further comprising, when the number of segments in the first audio signal that have voice activity dissatisfy the threshold value:

prompting, by the speech recognition-enabled electronic device, the user to speak the trigger phrase again; and

rejecting, by the speech recognition-enabled electronic device, the first audio signal corresponding to the user speaking the trigger phrase for use in training the trigger phrase model.

3. The computer-implemented method of claim 1, further comprising, for each frame in the received first audio signal:

identifying, by the speech recognition-enabled electronic device, audio characteristics of the user speaking the trigger phrase and audio characteristics of background noise for the corresponding frame in the received first audio signal;

comparing, by the speech recognition-enabled electronic device, the identified audio characteristics of the user speaking the trigger phrase to predetermined threshold values associated with one or more values for trigger phrase model training; and

determining, by the speech recognition-enabled electronic device, a voice activity detection flag for the corresponding frame in the received first audio signal in response to comparing the identified audio characteristics of the user speaking the trigger phrase to the predetermined threshold values.

4. The computer-implemented method of claim 3, wherein determining the voice activity detection flag for the corresponding frame in the received first audio signal comprises:

generating an accept enrollment flag in response to the identified audio characteristics of the user speaking the trigger phrase being less than the predetermined threshold values; and

generating a reject enrollment flag in response to the identified audio characteristics of the user speaking the trigger phrase being greater than the predetermined threshold values.

5. The computer-implemented method of claim 4, further comprising determining, by the speech recognition-enabled electronic device, a length of the user speaking the trigger phrase in the received first audio signal by counting a number of frames in the received first audio signal that comprise the accept enrollment flag.

6. The computer-implemented method of claim 5, further comprising:

comparing, by the speech recognition-enabled electronic device, the length of the segment of the user speaking the trigger phrase in the received first audio signal to a lower phrase length threshold and to a higher phrase length threshold; and

when the length of the user speaking the trigger phrase in the received first audio signal is less than the lower phrase length threshold or greater than the higher phrase length threshold, prompting, by the speech recognition-enabled electronic device, the user to speak the trigger phrase in a second attempt.

7. The computer-implemented method of claim 1, wherein the threshold value for the number of segments that have voice activity is based on a number of words or syllables in the trigger phrase determined during an offline analysis.

8. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

during a trigger phrase enrollment process;

prompting a user of the speech recognition-enabled electronic device to speak a trigger phrase;

receiving, at the speech recognition-enabled device, a first audio signal corresponding to the user speaking the trigger phrase;

determining a number of segments in the first audio signal that have voice activity, each segment comprising a corresponding sequence of continuous frames in the first audio signal that have voice activity; and

when the number of segments in the first audio signal that have voice activity satisfies a threshold value, training a trigger phrase model with the first audio signal corresponding to the user speaking the trigger phrase, the trigger phrase model configured to detect the trigger phrase in a spoken utterance; and

after the trigger phrase enrollment process;

receiving, while the speech recognition-enabled electronic device is in a sleep mode, a second audio signal including an utterance of the trigger phrase spoken by the user; and

detecting, using the trigger phrase model trained during the trigger phrase enrollment process, the utterance of the trigger phrase in the second audio signal, the trigger phrase when detected in the second audio signal causing the speech recognition-enabled electronic device to wake from the sleep mode, the sleep mode comprising a power-saving mode of operation in which one or more parts of the speech recognition-enabled electronic device are in a low-power state or powered off.

9. The system of claim 8, wherein the operations further comprise, when the number of segments in the first audio signal that have voice activity dissatisfy the threshold value:

prompting the user to speak the trigger phrase again; and

rejecting the first audio signal corresponding to the user speaking the trigger phrase for use in training the trigger phrase model.

10. The system of claim 8, wherein the operations further comprise, for each frame in the received first audio signal:

identifying audio characteristics of the user speaking the trigger phrase and audio characteristics of background noise for the corresponding frame in the received first audio signal;

comparing the identified audio characteristics of the user speaking the trigger phrase to predetermined threshold values associated with one or more values for trigger phrase model training; and

determining a voice activity detection flag for the corresponding frame in the received first audio signal in response to comparing the identified audio characteristics of the user speaking the trigger phrase to the predetermined threshold values.

11. The system of claim 10, wherein determining the voice activity detection flag for the corresponding frame in the received first audio signal comprises:

generating an accept enrollment flag in response to the identified audio characteristics of the user speaking the trigger phrase being less than the predetermined threshold values; and

generating a reject enrollment flag in response to the identified audio characteristics of the user speaking the trigger phrase being greater than the predetermined threshold values.

12. The system of claim 11, wherein the operations further comprise determining a length of the user speaking the trigger phrase in the received first audio signal by counting a number of frames in the received first audio signal that comprise the accept enrollment flag.

13. The system of claim 12, wherein the operations further comprise:

comparing the length of the segment of the user speaking the trigger phrase in the received first audio signal to a lower phrase length threshold and to a higher phrase length threshold; and

when the length of the user speaking the trigger phrase in the received first audio signal is less than the lower phrase length threshold or greater than the higher phrase length threshold, prompting the user to speak the trigger phrase in a second attempt.

14. The system of claim 8, wherein the threshold value for the number of segments that have voice activity is based on a number of words or syllables in the trigger phrase determined during an offline analysis.

15. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

during a trigger phrase enrollment process;

prompting a user of the speech recognition-enabled electronic device to speak a trigger phrase;

receiving, at the speech recognition-enabled device, a first audio signal corresponding to the user speaking the trigger phrase;

determining a number of segments in the first audio signal that have voice activity, each segment comprising a corresponding sequence of continuous frames in the first audio signal that have voice activity; and

when the number of segments in the first audio signal that have voice activity satisfies a threshold value, training a trigger phrase model with the first audio signal corresponding to the user speaking the trigger phrase, the trigger phrase model configured to detect the trigger phrase in a spoken utterance; and

after the trigger phrase enrollment process;

receiving, while the speech recognition-enabled electronic device is in a sleep mode, a second audio signal including an utterance of the trigger phrase spoken by the user; and

detecting, using the trigger phrase model trained during the trigger phrase enrollment process, the utterance of the trigger phrase in the second audio signal, the trigger phrase when detected in the second audio signal causing the speech recognition-enabled electronic device to wake from the sleep mode, the sleep mode comprising a power-saving mode of operation in which one or more parts of the speech recognition-enabled electronic device are in a low-power state or powered off.

16. The computer-readable medium of claim 15, wherein the operations further comprise, when the number of segments in the first audio signal that have voice activity dissatisfy the threshold value:

prompting the user to speak the trigger phrase again; and

rejecting the first audio signal corresponding to the user speaking the trigger phrase for use in training the trigger phrase model.

17. The computer-readable medium of claim 15, wherein the operations further comprise, for each frame in the received first audio signal:

identifying audio characteristics of the user speaking the trigger phrase and audio characteristics of background noise for the corresponding frame in the received first audio signal;

comparing the identified audio characteristics of the user speaking the trigger phrase to predetermined threshold values associated with one or more values for trigger phrase model training; and

determining a voice activity detection flag for the corresponding frame in the received first audio signal in response to comparing the identified audio characteristics of the user speaking the trigger phrase to the predetermined threshold values.

18. The computer-readable medium of claim 17, wherein determining the voice activity detection flag for the corresponding frame in the received first audio signal comprises:

generating an accept enrollment flag in response to the identified audio characteristics of the user speaking the trigger phrase being less than the predetermined threshold values; and

generating a reject enrollment flag in response to the identified audio characteristics of the user speaking the trigger phrase being greater than the predetermined threshold values.

19. The computer-readable medium of claim 18, wherein the operations further comprise determining a length of the user speaking the trigger phrase in the received first audio signal by counting a number of frames in the received first audio signal that comprise the accept enrollment flag.

20. The computer-readable medium of claim 19, wherein the further operations comprise:

comparing the length of the segment of the user speaking the trigger phrase in the received first audio signal to a lower phrase length threshold and to a higher phrase length threshold; and

when the length of the user speaking the trigger phrase in the received first audio signal is less than the lower phrase length threshold or greater than the higher phrase length threshold, prompting the user to speak the trigger phrase in a second attempt.