High-order entropy error functions for neural classifiers

• US 20050015251A1
• Filed: 05/08/2001
• Published: 01/20/2005
• Est. Priority Date: 05/08/2001
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Jia, Ying, Pi, Xiaobo
• Granted Patent: US 7,346,497 B2
• Time in Patent Office: Days
• US Class Current: 704/232
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/10 using distance or distortion measures between unknown speech and reference templates
• CPC Class Codes: G10L 15/16 using artificial neural networks

Abstract

An automatic speech recognition system comprising a speech decoder to resolve phone and world level information, a vector generator to generate information vectors on which a confidence measure is based by a neural network classifier (ANN). An error signal is designed which is not subject to false saturation or over specialization. The error signal is integrated into an error function which is back propagated through the ANN.

Claims

1. Artificial neural network classifier comprising:

a plurality of stages including an input layer, an output layer and at least a hidden layer intermediate said input layer and said output layer, each node comprising a summer, a function system to operate on a sum produced at said node, the function circuit providing an operator of

$\frac{{\left(2<br><br>t_j-1\right)}^{n-1}<br><br>{\left(t_j-y_j^L\right)}^n}{y_j^L<br><br>\left(1-y_j^L\right)}=d<br><br>y_j^L,$

where t_j is the value of an error signal at a node j, n is greater than or equal to two and L is the number of layers.

2. The artificial neural network according to claim 1 wherein each transform circuit provides the error function of claim 1 wherein the error function of said classifier is

$-\sum _{}^{}$

j = 1 L n ⁢

∫

o + 1 - ⁢

( 2 ⁢

t j - 1 ) n - 1 ⁢

( t j - y j L ) n y j L ⁡

( 1 - y j L ) = ⅆ

y j L .

3. The artificial neural network classifier of claim 2 wherein 2≦

n≦

3.

4. The artificial neural network classifier according to claim 3 wherein the error signal produced by said classifier designed to be of the form δ

_j^L=(2t_j−

1)ⁿ⁼¹(t_j−

y_j^L)ⁿ, where t_j is the value of an error signal node j and L is the number of layers.

10. The automatic speech recognition system according to claim 9 wherein each transform circuit provides the error function of claim 1 wherein the error function of said classifier is

$-\sum _{}^{}$

j = 1 L n ⁢

∫

o + 1 - ⁢

( 2 ⁢

t j - 1 ) n - 1 ⁢

( t j - y j L ) n y j L ⁡

( 1 - y j L ) = ⅆ

y j L .

5. An automatic speech recognition system comprising:

a speech decoder to decode phone level acoustic data and to collect and calculate feature vectors and further comprising feature vectors and adaptive neural network classifier to operate on the normalized feature vectors, said classifier comprising a circuit for transforming inputs thereto by a function

$-\underset{j=1}{\overset{L_n}{\sum <br><br>}}<br><br>{\int <br><br>}_{o^{-}}^{1^{-}}<br><br>\frac{{\left(2<br><br>t_j-1\right)}^{n-1}<br><br>{\left(t_j-y_j^L\right)}^n}{y_j^L<br><br>\left(1-y_j^L\right)}=d<br><br>y_j^L.$

6. The automatic speech recognition system according to claim 5 wherein said vector generator comprises a normalizing circuit for normalizing each feature vector according to

$x^{\prime }$

= { min x ≤

min ( x - min ) ( max - min ) <

x <

max max  

⁢

x ≥

max

where “

min” and

“

max”

are the minimum and maximum value of feature x respectively in a set of training data.

7. The automatic speech recognition system according to claim 6 wherein said feature vector generator circuit comprises calculation circuits to produce 8-dimensional feature vectors.

8. The automatic speech recognition system according to claim 6 wherein said 8 dimensions comprise A-stabil, LogA WE-end, NscoreQ, N-active-leafs, NScore, Score-per-frame, Duration.

9. The automatic speech recognition system according to claim 8 wherein said classifier comprises an MPL having nodes and wherein one node comprises a summer, a function system to operate on a sum produced at said node, the function circuit providing an operator of

$\frac{{(2}^{}}{}$

t j - 1 ) n - 1 ⁢

( t j - y j L ) n y j L ⁡

( 1 - y j L ) = ⅆ

y j L ,

where t_j is the value of an error signal at a node j, n is greater than or equal to two and L is the number of layers.

11. A method to generate an error function neural network classifier producing an output indicative of a confidence level for a decoded word comprising:

deriving an error signal as a linear function with respect to point pairs (t_j,1−

t_j) and (1−

t_j,t_j) as δ

_j^L=(2t_j−

1)ⁿ⁼¹(t_j−

y_j^L)ⁿ, where n=1 for a cross entropy function and raising die value of n to a higher order and back propagating the difference expression through the adaptive neural network classifier.

12. The method according to claim 11 wherein raising the difference expression to a higher order comprises setting 2≦

n>

3.

13. The method of claim 12 further comprising training the neural network in accordance with the function of claim 12.

14. A computer program product comprising computer usable medium having computer readable program code embodied in said medium to utilize output of a speech decoder and produce a multiple dimension output vector based on said output;

computer readable program code means for causing said computer to operate as a multiple layer perception classifier and computer readable program code means for causing said computer to ply a function to a sum at each node of the multi-layer perceptron of

$\frac{{\left(2<br><br>t_j-1\right)}^{n-1}<br><br>{\left(t_j-y_j^L\right)}^n}{y_j^L<br><br>\left(1-y_j^L\right)}=d<br><br>y_j^L.$

15. The computer program product according to claim 14 wherein said computer readable code comprises means for applying the function of claim 14 at each node to cause the multi-layer perception to produce the function

$-\sum _{}^{}$

j = 1 L o ⁢

∫

o + 1 - ⁢

( 2 ⁢

t j - 1 ) n - 1 ⁢

( t j - y j L ) n y j L ⁡

( 1 - y j L ) ⁢

ⅆ

y j L .

16. A computer program product according to claim 15 further comprising computer program means for normalizing each said dimension of said vector in accordance with the maximum and minimum values for each dimension over a set of training data.