Sound identification utilizing periodic indications

• US 10,460,723 B2
• Filed: 05/30/2018
• Issued: 10/29/2019
• Est. Priority Date: 02/24/2017
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Fukuda, Takashi, Ichikawa, Osamu, Ramabhadran, Bhuvana
• Primary Examiner(s): Colucci, Michael C
• Application Number:
  US15/992,778
• Publication Number:
  US 20180277104A1
• Time in Patent Office: 517 Days
• Field of Search: 704201, 704232, 704202, 706 15, 706 14
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: G10L 25/24 the extracted parameters being the cepstrum

Abstract

A computer-implemented method is provided. The computer-implemented method is performed by a speech recognition system having at least a processor. The method includes estimating sound identification information from a neural network having periodic indications and components of a frequency spectrum of an audio signal data inputted thereto. The method further includes performing a speech recognition operation on the audio signal data to decode the audio signal data into a textual representation based on the estimated sound identification information. The neural network includes a plurality of fully-connected network layers having a first layer that includes a plurality of first nodes and a plurality of second nodes. The method further comprises training the neural network by initially isolating the periodic indications from the components of the frequency spectrum in the first layer by setting weights between the first nodes and a plurality of input nodes corresponding to the periodic indications to 0.

Claims

1. A computer-implemented method performed by a speech recognition system having at least a processor, the method comprising:

estimating, by the processor, sound identification information from a neural network having periodic indications and components of a frequency spectrum of an audio signal data inputted thereto; and

performing, by the processor, a speech recognition operation on the audio signal data to decode the audio signal data into a textual representation based on the estimated sound identification information,

wherein the neural network includes a plurality of fully-connected network layers having a first layer that includes a plurality of first nodes and a plurality of second nodes, and wherein the method further comprises training the neural network by initially isolating the periodic indications from the components of the frequency spectrum in the first layer by setting weights between the first nodes and a plurality of input nodes corresponding to the periodic indications to 0.

2. The computer-implemented method of claim 1, wherein the estimating sound identification includes identifying phoneme information.

3. The computer-implemented method of claim 1, wherein the periodic indications represent fluctuations in the frequency spectrum that periodically appear in the frequency spectrum.

4. The computer-implemented method of claim 1, wherein the periodic indications represent harmonic structure of the audio signal data.

5. The computer-implemented method of claim 1, further comprising normalizing the periodic indications before the inputting into the neural network.

6. The computer-implemented method of claim 5, wherein the normalizing the periodic indications includes maintaining an ordinal scale among a plurality of bands in the periodic indications.

7. The computer-implemented method of claim 6, wherein the normalizing the periodic indications is based on sigmoid normalization or max-variance normalization.

8. The computer-implemented method of claim 1, wherein the components of the frequency spectrum include values relating to powers of the audio signal data in a plurality of frequency bands in the frequency spectrum.

9. The computer-implemented method of claim 8, wherein the periodic indications and the components of the frequency spectrum are inputted into the neural network by inputting a first derivation and a second derivation with respect to time of the values relating to powers of the audio signal data in the plurality of frequency bands in the frequency spectrum.

10. The computer-implemented method of claim 1, wherein the neural network is a convolutional neural network or a deep neural network.

11. The computer-implemented method of claim 10, wherein the periodic indications and the components of the frequency spectrum are input into a first layer of the neural network.

12. The computer-implemented method of claim 10, further comprising Mel-filtering the periodic indications and the frequency spectrum before the inputting into the neural network.

13. The computer-implemented method of claim 10, wherein the periodic indications are inputted into a second layer or a subsequent layer of the neural network.

14. The computer-implemented method of claim 13, wherein the neural network is the convolutional neural network, and the convolutional neural network includes one or more convolutional neural network layers, and

wherein the periodic indications are inputted into a layer that is downstream of the one or more convolutional neural network layers.

15. The computer-implemented method of claim 13, further comprising compressing the periodic indications by reducing a number of dimensions of the periodic indications before the inputting of the periodic indications into the neural network.

16. The computer-implemented method of claim 13, further comprising Mel-filtering the periodic indications before the periodic indications are inputted into the neural network.

17. A non-transitory computer program product having instructions embodied therewith, the instructions executable by a speech recognition system that includes a processor or programmable circuitry to cause the processor or programmable circuitry to perform a method comprising:

estimating sound identification information from a neural network having periodic indications and components of a frequency spectrum of an audio signal data inputted thereto; and

performing a speech recognition operation on the audio signal data to decode the audio signal data into a textual representation based on the estimated sound identification information,

wherein the neural network includes a plurality of fully-connected network layers having a first layer that includes a plurality of first nodes and a plurality of second nodes, and wherein the method further comprises training the neural network by initially isolating the periodic indications from the components of the frequency spectrum in the first layer by setting weights between the first nodes and a plurality of input nodes corresponding to the periodic indications to 0.

18. The non-transitory computer program product of claim 17, wherein the estimating sound identification includes identifying phoneme information.

19. The non-transitory computer program product of claim 17, wherein the periodic indications represent fluctuations in the frequency spectrum that periodically appear in the frequency spectrum.

20. A speech recognition system, comprising:

a processor; and

one or more computer readable mediums collectively including instructions that, when executed by the processor, cause the processor to;

estimate sound identification information from a neural network having periodic indications and components of a frequency spectrum of an audio signal data inputted thereto; and

perform a speech recognition operation on the audio signal data to decode the audio signal data into a textual representation based on the estimated sound identification information,

wherein the neural network is trained by initially isolating the periodic indications from the components of the frequency spectrum in the first layer by setting weights between the first nodes and a plurality of input nodes corresponding to the periodic indications to 0.