Method and system of neural network keyphrase detection

• US 10,650,807 B2
• Filed: 09/18/2018
• Issued: 05/12/2020
• Est. Priority Date: 09/18/2018
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Bocklet, Tobias, Ossowski, Jacek, Dorau, Tomasz, Muchlinski, Maciej, Pearce, David, Rozen, Piotr
• Primary Examiner(s): Hang, Vu B
• Application Number:
  US16/134,826
• Publication Number:
  US 20190043488A1
• Time in Patent Office: 602 Days
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/30 using neural networks
• CPC Class Codes: G10L 25/84 for discriminating voice from noise

Abstract

A method and system are directed to autonomous neural network keyphrase detection and includes generating and using a multiple element state score vector by using neural network operations and without substantial use of a digital signal processor (DSP) to perform the keyphrase detection.

Claims

1. A keyphrase detection device comprising:

memory storing received audio input; and

at least one neural network accelerator communicatively connected to the memory to receive the audio input and to operate by;

generating a multiple element acoustic score vector for a current time instance based on the received audio input, wherein the multiple element acoustic score vector comprises at least one acoustic score for at least one rejection model and acoustic scores for at least one multiple state keyphrase model, and wherein the multiple state keyphrase model corresponds to a predetermined keyphrase,

recursively aligning elements of a previous multiple element state score vector for a previous time instance with the elements of the multiple element acoustic score vector, wherein the previous multiple element state score vector comprises at least one previous state score for the single state rejection model and previous state scores for the multiple state keyphrase model,

generating an intermediate score vector by using scores from both the multiple element acoustic score vector and the previous multiple element state score vector,

generating a current multiple element state score vector of a current time instance comprising performing a propagating operation with the intermediate score vector, and

determining whether the received audio input is associated with the predetermined keyphrase by using scores from the current multiple element state score vector; and

a controller to provide at least one command to perform an action when the received audio input is associated with the predetermined keyphrase.

2. The device of claim 1 comprising a keyphrase detection unit autonomously operated by the network accelerator without using a digital signal processor to perform feature extraction, acoustic scoring, and keyphrase decoding by the neural network accelerator and without the use of a digital signal processor.

3. The device of claim 1 wherein the neural network accelerator generates a final score by using a first state score based on the single state rejection model and a second state score that is one of the state scores based on the keyphrase model and wherein both the first and second state scores are from the current multiple element state score vector;

wherein the generating of a final score comprises operating an affine neural network layer wherein the first and second state scores are inputs to the affine layer;

wherein scores on the current multiple element state score vector in addition to the first and second state scores are input to the affine layer and are provided a weight of zero;

wherein the final score is computed by using a threshold value as an input of the affine layer;

wherein the threshold value has a weight of −

1, the first state score has a weight of −

1, and the second state score has a weight of 1.

4. The device of claim 1 wherein the neural network accelerator operates as if performing a recurrent neural network layer comprising providing a just generated current multiple element state score vector as a previous multiple element state score vector to be used with a next multiple element acoustic score vector to generate the vector of intermediate state score values.

5. The device of claim 1 wherein the neural network accelerator treats elements of the previous multiple element state score vector as bias values to be applied to elements of the multiple element acoustic score vector.

6. The device of claim 1 wherein the propagating operation is a determination of a maximum value among values of individual adjacent pairs of states along the intermediate score vector to form the current multiple element state score vector.

7. The device of claim 1 wherein the neural network accelerator generates the at least one acoustic score for at least one rejection model by separately pooling multiple acoustic base scores of multiple rejection categories of silence, phonetics, speech noise and non-speech noise to form a single category score of each category before pooling the single category scores together to form a single acoustic rejection score.

8. A computer-implemented system for performing keyphrase detection comprising:

at least one microphone to capture audio input;

memory to store the audio input;

at least one processor communicatively coupled to the at least one microphone and at least one memory, and comprising at least one neural network accelerator to receive the audio input and operate by;

generating a multiple element acoustic score vector for a current time instance based on received audio input, wherein the multiple element acoustic score vector comprises at least one acoustic score for at least one rejection model and acoustic scores for at least one multiple state keyphrase model, and wherein the multiple state keyphrase model corresponds to a predetermined keyphrase;

recursively aligning elements of a previous multiple element state score vector for a previous time instance with the elements of the multiple element acoustic score vector, wherein the previous multiple element state score vector comprises at least one previous state score for the single state rejection model and previous state scores for the multiple state keyphrase model;

generating an intermediate score vector by using scores from both the multiple element acoustic score vector and the previous multiple element state score vector,

generating a current multiple element state score vector of a current time instance comprising performing a propagating operation with the intermediate score vector, and

determining whether current state scores of the current multiple element state score vector indicate that the received audio input is associated with the predetermined keyphrase; and

providing at least one command to perform an action when the received audio input is associated with the predetermined keyphrase.

9. The system of claim 8, wherein the neural network accelerator generates both a speech final score and a non-speech final score comprising pooling at least some of the same base acoustic rejection scores used to generate the at least one acoustic score for the at least one rejection model;

and

using the speech and non-speech final scores to determine whether speech or non-speech exists in the audio input.

10. The system of claim 8 wherein the neural network accelerator is to operate by forming a multiple element vector of pooled rejection acoustic scores by using the base acoustic scores;

and performing operations on the multiple element vector of pooled rejection acoustic scores to form a speech final score and a non-speech final score.

11. The system of claim 10 wherein at least one speech-related score of the multiple element vector of pooled rejection acoustic scores is based on the pooling of speech-related base rejection scores and at least one non-speech-related score of the multiple element vector of pooled rejection acoustic scores is based on the pooling of non-speech-related base rejection scores.

12. The system of claim 11, wherein the base rejection scores of phonetics and speech-related noise rejection categories are pooled together to form the speech-related scores, and silence and non-speech noise categories are pooled together to form the non-speech-related scores.

13. The system of claim 10, wherein the pooled rejection acoustic scores to be used as elements in the multiple element vector of pooled rejection acoustic scores are pooled together to form the at least one acoustic score of the at least one rejection model on the multiple element acoustic score vector to detect a keyphrase.

14. The system of claim 13 wherein the first and last score of the multiple element vector of pooled rejection acoustic scores are based on the pooling of both the same of speech or non-speech-related base rejection scores and the acoustic rejection scores between the first and last scores are pooled rejection acoustic scores based on the other of the speech or non-speech-related base rejection scores.

15. The system of claim 14 wherein the first and second scores of the current multiple element rejection state score vector is used to form one of the speech or non-speech final score, and the last and second to last scores of the current multiple element rejection state score vector are used to form the other of the speech or non-speech final score.

16. The system of claim 14 wherein scores of the current multiple element rejection state score vector are input to an affine neural network layer operation to form the speech and non-speech final scores;

and wherein the weights of −

1 are provided to one of the non-speech and speech scores of the current multiple element rejection state score vector and a weight of 1 to the other of the speech or non-speech scores of the current multiple element rejection state score vector.

17. The system of claim 9 wherein the operations are the same operations as performed on the multiple element acoustic score vector to form a current multiple element rejection state score vector to be used to form the speech and non-speech final scores.

18. The system of claim 9 wherein the neural network accelerator operates by at least one of:

concatenating a plurality of multiple element acoustic score vectors each of a different keyphrase, and separately analyzing parallel multiple element acoustic score vectors each of a different keyphrase.

19. At least one non-transitory machine readable medium comprising a plurality of instructions that, in response to being executed on a computing device, cause the computing device to perform keyphrase detection by:

generating, by a neural network accelerator, a multiple element acoustic score vector for a current time instance based on received audio input, wherein the multiple element acoustic score vector comprises at least one acoustic score for at least one rejection model and acoustic scores for at least one multiple state keyphrase model, and wherein the multiple state keyphrase model corresponds to a predetermined keyphrase;

recursively aligning elements of a previous multiple element state score vector for a previous time instance with the elements of the multiple element acoustic score vector, wherein the previous multiple element state score vector comprises at least one previous state score for the single state rejection model and previous state scores for the multiple state keyphrase model;

generating, by a neural network accelerator, an intermediate score vector by using scores from both the multiple element acoustic score vector and the previous multiple element state score vector;

generating, by a neural network accelerator, a current multiple element state score vector of a current time instance comprising performing a propagating operation with the intermediate score vector;

determining, by a neural network accelerator, whether current state scores of the current multiple element state score vector indicate that the received audio input is associated with the predetermined keyphrase; and

providing at least one command to perform an action when the received audio input is associated with the predetermined keyphrase.

20. The machine readable medium of claim 19, wherein the instructions cause the computing device to operate by using at least one booling operation to perform keyphrase start-point detection at the neural network accelerator.

21. The machine readable medium of claim 20, wherein forming the first bool vector comprises:

comparing adjacent elements of a keyphrase score vector output from a keyphrase detection operation,

applying −

1, 0, and 1 weights at nodes of a neural network to the results of the comparisons, and

applying a non-linear activation function to the weighted combinations to obtain a sequence of 0s and 1s.

22. The machine readable medium of claim 20 wherein the instructions cause the computing device to operate by:

forming a first bool vector at least partially based on a keyphrase score vector output from a keyphrase detection operation, and forming a second inverse bool vector that is an inverse of the first bool vector;

separately updating a previous output vector of frame index numbers with the first and inverse bool vectors to respectively form first and second updated vectors; and

combining the first and second updated vectors to form a current output vector of frame index numbers wherein a frame index number at a specific element location on the current output vector indicates a start point of a keyphrase.

23. The machine readable medium of claim 22 wherein the updated vectors are formed by performing element-by-element multiplication between elements of one of the bool vectors and elements of the previous output vector.

24. The machine readable medium of claim 19, wherein the instructions cause the computing device to modify a buffer size to hold the audio input depending on whether speech or non-speech was detected in the audio input as indicated by the neural network accelerator.

25. The machine readable medium of claim 19, wherein the instructions cause the computing device to modify the number of active microphones of a plurality of active microphones depending, at least in part, on noise rejection scores received by the neural network accelerator.