Neural network method and apparatus

• US 10,504,506 B2
• Filed: 12/07/2017
• Issued: 12/10/2019
• Est. Priority Date: 08/14/2017
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Yoo, Sang Hyun
• Primary Examiner(s): Patel, Yogeshkumar
• Application Number:
  US15/834,416
• Publication Number:
  US 20190051291A1
• Time in Patent Office: 733 Days
• Field of Search: 704233, 704246
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/20 Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech G10L21/02 takes precedence
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A neural network method and apparatus, the method including providing a voice signal to a main neural network and a sub-neural network, obtaining a scaling factor by implementing the sub-neural network configured to generate the scaling factor by interpreting the provided voice signal, determining a size of a future context, based on the scaling factor, to be considered by the main neural network configured to perform speech recognition, obtaining a result of a recognizing of the voice signal by implementing the main neural network with the determined size of the future context.

Claims

1. A processor implemented recognition method, comprising:

providing a voice signal to a main neural network and a sub-neural network;

obtaining a scaling factor by implementing the sub-neural network configured to generate the scaling factor by interpreting the provided voice signal;

determining a size of a future context, based on the scaling factor, to be considered by the main neural network configured to perform speech recognition; and

obtaining a result of a recognizing of the voice signal by implementing the main neural network with the determined size of the future context.

2. The recognition method of claim 1, wherein the main neural network includes a first layer having previous time connections and a second layer having future time connections.

3. The recognition method of claim 1, wherein the determining of the size of the future context includes determining sizes of respective future contexts for respective speech recognitions by the main neural network, the sizes of the respective future contexts being dynamically determined dependent on respective context interpretations of voice signals by the sub-neural network.

4. The recognition method of claim 1, wherein the providing of the voice signal to the main neural network includes providing plural audio frames or windows to the main neural network, and the providing of the voice signal to the sub-neural network includes providing less than all of the plural audio frames or windows to the sub-neural network,

wherein the sub-neural network is configured to generate the scaling factor by interpreting the provided less than all of the plural audio frames or windows, and the main neural network performs the recognizing of the voice signal based on the plural audio frames or windows.

5. The recognition method of claim 1, wherein, for the obtaining of the scaling factor, the providing of the voice signal to the sub-neural network includes selecting between providing all of the plural audio frames or windows to the sub-neural network and providing less than all of the plural audio frames or windows to the sub-neural network.

6. The recognition method of claim 1, wherein the obtaining of the scaling factor comprises obtaining, using the sub-neural network, the scaling factor as indicating a time based significance of the future context to be considered by the main neural network when recognizing the voice signal.

7. The recognition method of claim 6, wherein the obtaining of the scaling factor comprises obtaining the scaling factor based a trained interpretation by the sub-neural network of a determined or interpreted level of noise included in the voice signal and/or a determined or interpreted accuracy of user pronunciation included in the voice signal.

8. The recognition method of claim 6, wherein the obtaining of the scaling factor includes generating the scaling factor with a relatively large number of components having values greater than a predetermined threshold based on a clarity of the voice signal being interpreted to be relatively low by the sub-neural network, such that the size of the future context increases as a number of components that are greater than the predetermined threshold increase and decreases as the number of the components that are greater than the predetermined threshold decrease.

9. The recognition method of claim 1, wherein the determining of the size of the future context includes:

determining, based on the scaling factor, the size of the future context to be greater as a clarity of the voice signal is determined or interpreted to be relatively low by the sub-neural network than as the clarity of the voice signal is determined or interpreted to be relatively high by the sub-neural network, and

determining, based on the scaling factor, the size of the future context to be smaller or zero as the clarity of the voice signal is interpreted to be relatively high by the sub-neural network than as the clarity of the voice signal is determined or interpreted to be relatively low by the sub-neural network.

10. The recognition method of claim 1, wherein the determining of the size of the future context includes determining the size of the future context by comparing values of components of the scaling factor to a predetermined threshold.

11. The recognition method of claim 10, wherein the determining of the size of the future context further includes determining the size of the future context based on a determined highest dimension component, among all components of the scaling factor, having a value greater than the predetermined threshold.

12. The recognition method of claim 1, wherein the obtaining of the result of recognizing of the voice signal comprises:

adjusting, based on the determined size of the future context, an extent among plural future context for provision to a lookahead convolution layer of the main neural network;

implementing the main neural network based on the adjusted extent; and

obtaining the result of the recognizing of the voice signal from the lookahead convolution layer implemented based on the adjusted extent.

13. The recognition method of claim 12, wherein the adjusting of the extent among the plural future context for the provision to the lookahead convolution layer includes selectively, based on the determined size of the future context, providing between zero future context to the lookahead convolution layer, some future context but not all of the plural future context to the lookahead convolution layer, and all of the plural future context to the lookahead convolution layer.

14. The recognition method of claim 1, wherein the obtaining of the scaling factor by implementing the sub-neural network includes the sub-neural network generating the scaling factor from only a first window of plural windows of the voice signal, and

the obtaining of the result of recognizing of the voice signal includes obtaining the recognition result with respect to each of the plural windows.

15. The recognition method of claim 1, wherein the main neural network is a unidirectional recurrent neural network (RNN) including a lookahead convolution layer.

16. The recognition method of claim 1, wherein the main neural network and the sub-neural network are neural networks having been trained simultaneously.

17. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1.

18. A processor implemented training method, comprising:

obtaining a scaling factor from a sub-neural network into which a training input is input;

determining a size of a future context to be considered by a main neural network based on the scaling factor; and

training the main neural network and the sub-neural network such that the main neural network outputs a training output mapped to the training input in response to the training input being input into the main neural network to which the determined size of the future context is applied.

19. The training method of claim 18, wherein the training comprises training the main neural network and the sub-neural network simultaneously.

20. A recognition apparatus, comprising:

a processor configured to

provide a voice signal to a main neural network and a sub-neural network,

obtain a scaling factor by implementing the sub-neural network configured to generate the scaling factor by interpreting the provided voice signal,

determine a size of a future context, based on the scaling factor, to be considered by the main neural network configured to perform speech recognition, and

obtain a result of a recognizing of the voice signal by implementing the main neural network with the determined size of the future context.

21. The recognition apparatus of claim 20, further comprising a memory, the memory including instructions, that when executed by the processor, cause the processor to perform the providing of the voice signal, the obtaining of the scaling factor, the determining of the size of the future context, and the obtaining of the result of the recognizing.

22. The recognition apparatus of claim 20, further comprising a memory storing respective trained parameters of the main neural network and the sub-neural network, and the processor is further configured to apply the respective trained parameters when respectively implementing the main neural network and implementing the sub-neural network.

23. The recognition apparatus of claim 20, wherein, for the obtaining of the scaling factor, the processor is configured to obtain, using the sub-neural network, the scaling factor as indicating a time based significance of the future context to be considered by the main neural network when recognizing the voice signal.

24. The recognition apparatus of claim 23, wherein, for the obtaining of the scaling factor, the processor is configured to obtain the scaling factor based on a trained interpretation by the sub-neural network of a determined or interpreted level of noise included in the voice signal and/or a determined or interpreted accuracy of user pronunciation included in the voice signal.

25. The recognition apparatus of claim 20, wherein, for the determining of the size of the future context, the processor is configured to:

determine, based on the scaling factor, the size of the future context to be greater as a clarity of the voice signal is determined or interpreted to be relatively low by the sub-neural network than as the clarity of the voice signal is determined or interpreted to be relatively high by the sub-neural network, and

determine, based on the scaling factor, the size of the future context to be smaller or zero as the clarity of the voice signal is interpreted to be relatively high by the sub-neural network than as the clarity of the voice signal is determined or interpreted to be relatively low by the sub-neural network.

26. The recognition apparatus of claim 20, wherein, for the determining of the size of the future context, the processor is configured to determine the size of the future context by comparing values of components included in the scaling factor to a predetermined threshold.

27. The recognition apparatus of claim 20, wherein the recognition apparatus is an electronic device and further comprises a speech receiver configured to capture audio and the processor is configured to extract the voice signal from the captured audio,

wherein the processor is further configured to control non-speech recognition functions of the recognition apparatus based on interpreted commands indicated by the obtained results of the recognizing.

28. An electronic device, the electronic device comprising, comprising:

a speech receiver configured to capture audio;

a memory storing trained parameters of a main neural network and trained parameters of a sub-neural network; and

a processor configured to apply the trained parameters of the sub-neural network to implement the sub-neural network to which a voice signal derived from the captured audio is input, and to apply the trained parameters of the main neural network to implement the main neural network, to recognize the voice signal, with dynamically adjusted future context consideration based on results of the implementation of the sub-neural network,

wherein the main neural network includes a recurrent layer configured to consider past context.

29. The electronic device of claim 28, wherein the trained parameters of the sub-neural network configure the sub-neural network to interpret a clarity of the voice signal, with the dynamic adjusted future context consideration of the main neural network being performed by maximizing a size of the future context consideration of the main neural network when the sub-neural network interprets the clarity to be low and minimizing a size of the future context consideration of the main neural network when the sub-neural network interprets the clarity to be high.

30. The electronic device of claim 28, wherein the processor is further configured to indicate the result of the recognizing of the voice signal.