Method of optimising the execution of a neural network in a speech recognition system through conditionally skipping a variable number of frames

• US 7,769,580 B2
• Filed: 12/23/2002
• Issued: 08/03/2010
• Est. Priority Date: 12/23/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Loquendo SpA (Microsoft Corporation)
• Inventors: Albesano, Dario, Gemello, Roberto
• Primary Examiner(s): Dorvil; Richemond
• Assistant Examiner(s): YEN, ERIC L
• Application Number:
  US10/538,876
• Publication Number:
  US 20060111897A1
• Time in Patent Office: 2,780 Days
• Field of Search: 704/202, 704231-232
• US Class Current: 704/202
• CPC Class Codes: G10L 15/16 using artificial neural networks

Abstract

A method of optimizing the execution of a neural network in a speech recognition system provides for conditionally skipping a variable number of frames, depending on a distance computed between output probabilities, or likelihoods, of a neural network. The distance is initially evaluated between two frames at times 1 and 1+k, where k is a predetermined maximum distance between frames, and if such distance is sufficiently small, the frames between times 1 and 1+k are calculated by interpolation, avoiding further executions of the neural network. If, on the contrary, such distance is not small enough, it means that the outputs of the network are changing quickly, and it is not possible to skip too many frames. In that case, the method attempts to skip remaining frames, calculating and evaluating a new distance.

Claims

1. A method of executing a neural network in a speech recognition system for recognizing speech of an input speech signal organized into a series of frames, comprising:

calculating, by means of said neural network, a first and a second likelihood corresponding to a first and a second non-consecutive frame;

calculating a distance between said first and second non-consecutive frames;

comparing said distance with a predetermined threshold value to evaluate a possibility of skipping at least one run of the neural network;

selectively skipping the at least one run of the neural network in correspondence to each frame between said first and said second non-consecutive frames to optimize the neural network when said distance is lower than said threshold value;

calculating a likelihood or likelihoods corresponding to each frame between said first and second non-consecutive frames;

calculating said distance as a distance between output likelihoods of said neural network; and

providing an optimized neural network by outputting the likelihood or likelihoods corresponding to each frame between said first and second non-consecutive frames to a computer readable medium.

2. The method according to claim 1, further comprising:

a) buffering a plurality of input frames;

b) defining an interval corresponding initially to a main interval of frames delimited by a first and a second non-consecutive buffered frames;

c) calculating, by means of said neural network, said first and said second likelihood corresponding to said buffered frames delimiting said interval;

d) calculating a distance between said first and said second likelihoods;

e) comparing said distance between said first and said second likelihoods with a threshold value and, in case said distance between said first and said second likelihoods is lower than said threshold value, calculating by interpolation between said first and said second likelihoods, the likelihood or likelihoods corresponding to each frame between said first and said second nonconsecutive buffered frames, or, in case said distance between said first and said second likelihoods is greater than said threshold value, calculating, by means of said neural network, at least one other likelihood corresponding to at least one other frame within said interval; and

f) applying recursively said steps c) to e) to each interval present as a sub-interval within said main interval containing at least one frame whose likelihood has not been yet calculated, until all the likelihoods corresponding to the frames in said main interval have been calculated.

3. The method as claimed in claim 2, wherein said interpolation is a linear interpolation.

4. The method as claimed in claim 2, wherein said main interval of frames comprises said plurality of buffered input frames.

5. The method as claimed in claim 2, wherein said likelihoods are probability distributions.

6. The method as claimed in claim 5, wherein said distance between said first and said second likelihoods is calculated as a symmetric Kullback distance between probability distributions.

7. The method as claimed in claim 2, wherein said threshold value is comprised of a fuzzy set.

8. The method as claimed in claim 7, wherein said fuzzy set has a domain corresponding to a percentage of output units of said neural network used by a current phonetic variability.

9. The method as claimed in claim 8, wherein said fuzzy set is a linear segmented decreasing function.

10. A speech recognition system for recognizing speech of an input speech signal, comprising:

a neural network for calculating likelihoods corresponding to frames of said input speech signal, comprising;

a buffer for storing a plurality of input frames;

a distance evaluation unit for calculating a distance between a first and a second likelihood, said first and second likelihoods being obtained by means of said neural network and corresponding to a first and a second non-consecutive buffered frames;

a comparing unit for comparing said distance with a predetermined threshold value to evaluate a possibility of skipping at least one run of the neural network; and

an interpolation unit for, after the comparing, in case said distance is lower than said threshold value, skipping a run of the neural network corresponding to each of the frame or frames between said first and second non-consecutive frames to optimize the neural network, and calculating the likelihood or likelihoods corresponding to the frame or frames between said first and second non-consecutive buffered frames, and

a computer readable medium for storing at least one output of the neural network, the at least one output comprising the likelihood or likelihoods corresponding to the frame or frames between said first and second non-consecutive buffered frames.

11. The speech recognition system according to claim 10, wherein said buffer is a lookahead buffer.

12. A non-transitory computer-readable medium for use on a computing system, the non-transitory computer-readable medium including computer-executable instructions for performing a method of executing a neural network in a speech recognition system for recognizing speech of an input speech signal organized into a series of frames, the method comprising:

calculating, by means of said neural network, a first and a second likelihood corresponding to a first and a second non-consecutive frame;

calculating a distance between said first and second non-consecutive frames;

comparing said distance with a predetermined threshold value to evaluate a possibility of skipping at least one run of the neural network;

selectively skipping the at least one run of the neural network in correspondence to each frame between said first and said second non-consecutive frames to optimize the neural network when said distance is lower than said threshold value;

calculating a likelihood or likelihoods corresponding to each frame between said first and second non-consecutive frames;

calculating said distance as a distance between output likelihoods of said neural network; and

providing an optimized neural network by outputting the likelihood or likelihoods corresponding to each frame between said first and second non-consecutive frames to the computer readable medium.

13. The non-transitory computer-readable medium of claim 12, wherein the method further comprises:

a) buffering a plurality of input frames;

b) defining an interval corresponding initially to a main interval of frames delimited by a first and a second non-consecutive buffered frames;

c) calculating, by means of said neural network, said first and said second likelihood corresponding to said buffered frames delimiting said interval;

d) calculating a distance between said first and said second likelihoods;

e) comparing said distance between said first and said second likelihoods with a threshold value and, in case said distance between said first and said second likelihoods is lower than said threshold value, calculating by interpolation between said first and said second likelihoods, the likelihood or likelihoods corresponding to each frame between said first and said second nonconsecutive buffered frames, or, in case said distance between said first and said second likelihoods is greater than said threshold value, calculating, by means of said neural network, at least one other likelihood corresponding to at least one other frame within said interval; and

f) applying recursively said steps c) to e) to each interval present as a sub-interval within said main interval containing at least one frame whose likelihood has not been yet calculated, until all the likelihoods corresponding to the frames in said main interval have been calculated.

14. The non-transitory computer-readable medium of claim 13, wherein said interpolation is a linear interpolation.

15. The non-transitory computer-readable medium of claim 13, wherein said main interval of frames comprises said plurality of buffered input frames.

16. The non-transitory computer-readable medium of claim 13, wherein said likelihoods are probability distributions.

17. The non-transitory computer-readable medium of claim 16, wherein said distance between said first and said second likelihoods is calculated as a symmetric Kullback distance between probability distributions.

18. The non-transitory computer-readable medium of claim 13, wherein said threshold value is comprised of a fuzzy set.

19. The non-transitory computer-readable medium of claim 18, wherein said fuzzy set has a domain corresponding to a percentage of output units of said neural network used by a current phonetic variability.

20. The non-transitory computer-readable medium of claim 19, wherein said fuzzy set is a linear segmented decreasing function.