Neural network for maximum likelihood classification with supervised and unsupervised training capability

• US 5,724,487 A
• Filed: 07/07/1995
• Issued: 03/03/1998
• Est. Priority Date: 07/07/1995
• Status: Expired due to Fees

• Current Assignee: Roy L. Streit
• Original Assignee: Roy L. Streit
• Inventors: Streit, Roy L.
• Primary Examiner(s): Hafiz, Tariq R.
• Application Number:
  US08/499,336
• Time in Patent Office: 970 Days
• Field of Search: 395/20-27, 382/155-159
• US Class Current: 706/25
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology

Abstract

A neural network comprising an input layer, two hidden layers for generating an number of outcome class component values, and an output layer for classifying input vectors to an outcome class, under the assumption that the outcome classes are characterized by mixtures of component populations with each component population having a multivariate Gaussian likelihood distribution. The first hidden layer includes a number of first layer nodes each connected receive input vector components from the input layer and generates in response a first layer output value representing the absolute value of the sum of a function of the difference between each input vector component and a threshold value. The second hidden layer includes a plurality of second layer nodes each for generating an outcome class component value, each second layer node being connected to predetermined ones of the first layer nodes and generating in response to the first layer output values an outcome class component value representing a function related to the exponential of the negative square of the sum of first layer output values connected thereto. The output layer includes a plurality of output layer nodes each associated with an outcome class. Each output layer node uses the output class component values from the second layer nodes in combination with weighting values to generate the likelihood that the input vector is properly classified to the output layer node'"'"'s outcome class.

Claims

1. A neural network for classifying input vectors to one of a plurality of outcome classes comprising:

an input layer comprising a plurality of input terminals each for receiving a component of an input vector;

two hidden layers, including (i) a first hidden layer having "H" first layer nodes, each being connected to the input terminals for receiving the input vector components from the input terminals and for generating a first layer output value representing the absolute value of the sum of the difference between a function of each input vector component and a threshold value, wherein the threshold value is determined as

space="preserve" listing-type="equation">t.sub.i =uT L-T s.sub.i,

where "u" is a mean vector of training input vectors used to train the neural network during the training operation, "L" is a Choleski factor of a covariance matrix for the training input vectors, "T" and "-T" represent a transpose and inverse transpose operation, respectively, and "s_i " represents a vector defining a surface point on an H-component grid defined on a unit sphere having a number of dimensions corresponding to the number of components of an input vector, and (ii) a second hidden layer including a plurality of second layer nodes each for generating an outcome class component value, each second layer node being connected to predetermined ones of the first layer nodes and generating in response to the first layer output values an outcome class component value representing a function related to the exponential of the negative square of the sum of first layer output values from the first layer nodes connected thereto; and

an output layer comprising a plurality of output nodes each associated with an outcome class, each output node receiving a plurality of outcome class component values, each associated with an outcome class component and each representing a contribution of the outcome class component to the outcome class determination, each output node performing a summation operation in connection with the value of product of each outcome class component and an associated outcome class component weighting value to generate an outcome class value, the collection of outcome class values from all of the output nodes identifying the outcome class associated with the input vector.

2. A neural network as defined in claim 1 in which the function of each input vector component applied by the "j-th" first layer node ("j" is an integer from "1" to "H") to an input vector component value received from the "i-th" input terminal ("i" is an integer from "1" to "N") in said first layer is the "j-th" component of

3. A neural network for classifying input vectors to one of a plurality of outcome classes comprising:

an input layer comprising a plurality of input terminals each for receiving a component of an input vector;

two hidden layers, including (i) a first hidden layer including a plurality of first layer nodes each connected to the input terminals for receiving the input vector components from the input terminals and for generating a first layer output value according to a first layer non-linear function representing the absolute value of the sum of the difference between a function of each input vector component and a threshold value, and (ii) a second hidden layer including a plurality of second layer nodes each for generating an outcome class component value, each second layer node being connected to predetermined ones of the first layer nodes and generating in response to the first layer output values an outcome class component value according to a second layer non-linear function related to the exponential of the negative square of the sum of first layer output values from the first layer nodes connected thereto, wherein the outcome class component value for a second layer node conforms to the likelihood p(X) of a component class mixture given by ##EQU26## where N is the number of input terminals, u is a mean vector of training input vectors, CM is a covariance matrix, L is a Cholesky factor of CM, T identifies a transpose, s_i is a vector defining a surface point on an H component grid defined on a unit sphere of N dimensions, V_N-1 is the volume of an N-1 dimension sphere, DEL is the surface area of an N-dimensional unit sphere divided by H, H is the number of surface elements on the N-dimensional unit sphere; and

an output layer comprising a plurality of output nodes each associated with an outcome class, each output node receiving a plurality of outcome class component values, each associated with an outcome class component and each representing a contribution of the outcome class component to the outcome class determination, each output node performing a summation operation in connection with the value of product of each outcome class component and an associated outcome class component weighting value to generate an outcome class value, the collection of outcome class values from all of the output nodes identifying the outcome class associated with the input vector.

4. A neural network as defined in claim 3 in which the outcome class component values generated by the second layer nodes reflects a threshold value of zero.

5. A neural network as defined in claim 3 in which the second layer nodes all apply the same weighting value in connection with first layer output values they receive from the first layer nodes in generating the respective outcome class component values.

6. A neural network as defined in claim 3 in which the outcome class value generated by each output node in the output level reflects a threshold value of zero.

7. A neural network as defined in claim 3, trained in accordance with a supervised training operation in which a plurality of training input vectors each of known outcome class is applied to the neural network and threshold values and weighting values of said first layer nodes, and weighting values of said output nodes are determined to identify the outcome class for each training input vector.

8. A neural network for classifying input vectors to one of a plurality of outcome classes comprising:

an input layer comprising a plurality of input terminals each for receiving a component of an input vector;

two hidden layers, including (i) a first hidden layer including a plurality of first layer nodes each connected to the input terminals for receiving the input vector components from the input terminals and for generating a first layer output value representing the absolute value of the sum of the difference between a function of each input vector component and a threshold value, and (ii) a second hidden layer including a plurality of second layer nodes each for generating an outcome class component value, each second layer node being connected to predetermined ones of the first layer nodes and generating in response to the first layer output values an outcome class component value representing a function related to the exponential of the negative square of the sum of first layer output values from the first layer nodes connected thereto; and

an output layer comprising a plurality of output nodes each associated with an outcome class, each output node receiving a plurality of outcome class component values, each associated with an outcome class component and each representing a contribution of the outcome class component to the outcome class determination, each output node performing a summation operation in connection with the value of product of each outcome class component and an associated outcome class component weighting value to generate an outcome class value, the collection of outcome class values from all of the output nodes identifying the outcome class associated with the input vector; and

trained in accordance with an unsupervised training operation in which a plurality of sets of training input vectors, each of unknown outcome class, related within each set by a predetermined parameter, are applied to the input terminals and the threshold values and weighting values of said first layer nodes and weighting values of said output nodes are determined so that each set of training input vectors is a sequence of a hidden Markov model and each outcome class represented by one of said output nodes is a state of the hidden Markov model.

9. A method of training a neural network, the neural network classifying input vectors to one of a plurality of outcome classes, under the assumption that the outcome classes are characterized by mixtures of component populations with each component population having a multivariate Gaussian likelihood distribution, the neural network comprising an input layer comprising a plurality of input terminals each for receiving an component of an input vector, two hidden layers, including (i) a first hidden layer including a plurality of first layer nodes each connected to the input terminals for receiving the input vector components from the input terminals and for generating a first layer output value representing the absolute value of the sum of the difference between a function of each input vector component and a threshold value, and (ii) a second hidden layer including a plurality of second layer nodes each for generating an outcome class component value, each second layer node being connected to predetermined ones of the first layer nodes and generating in response to the first layer output values an outcome class component value representing a function related to the exponential of the negative square of the sum of first layer output values from the first layer nodes connected thereto;

and an output layer comprising a plurality of output nodes each associated with an outcome class, each output node receiving a plurality of outcome class component values, each associated with an outcome class component and each representing a contribution of the outcome class component to the outcome class determination, each output node performing a summation operation in connection with the value of product of each outcome class component and an associated outcome class component weighting value to generate an outcome class value, the collection of outcome class values from all of the output nodes identifying the outcome class associated with the input vector, the method comprising the steps of;

selecting a value for "K" corresponding to the number of second layer nodes;

applying a plurality of training input vectors each of known outcome class to the neural network to generate threshold values and weighting values of said first and second layer nodes;

determining a value corresponding to the number of correct classifications by the neural network of the training input vectors; and

performing the selecting, applying and determining steps through a plurality of iterations each with a different value for "K" and selecting the neural network with a value for "K" and the threshold and weighting values determined which provide the optimal classification.

10. A method of training a neural network, the neural network classifying input vectors to one of a plurality of outcome classes, under the assumption that the outcome classes are characterized by mixtures of component populations with each component population having a multivariate Gaussian likelihood distribution, the neural network comprising an input layer comprising a plurality of input terminals each for receiving an component of an input vector, two hidden layers, including (i) a first hidden layer including a plurality of first layer nodes each connected to the input terminals for receiving the input vector components from the input terminals and for generating a first layer output value representing the absolute value of the sum of the difference between a function of each input vector component and a threshold value, and (ii) a second hidden layer including a plurality of second layer nodes each for generating an outcome class component value, each second layer node being connected to predetermined ones of the first layer nodes and generating in response to the first layer output values an outcome class component value representing a function related to the exponential of the negative square of the sum of first layer output values from the first layer nodes connected thereto;

and an output layer comprising a plurality of output nodes each associated with an outcome class, each output node receiving a plurality of outcome class component values, each associated with an outcome class component and each representing a contribution of the outcome class component to the outcome class determination, each output node performing a summation operation in connection with the value of product of each outcome class component and an associated outcome class component weighting value to generate an outcome class value, the collection of outcome class values from all of the output nodes identifying the outcome class associated with the input vector, the method comprising the steps of;

selecting a value for "K" corresponding to the second of second layer nodes;

applying a plurality of sets of training input vectors each of unknown outcome class, the training input vectors being related within each set by a predetermined parameter, so that each set of training input vectors is a sequence of a hidden Markov model and each outcome class represented by one of said output nodes is a state of the hidden Markov model to the neural network to generate threshold values and weighting values of said first and second layer nodes;

determining a value corresponding to the number of correct classifications by the neural network of the training input vectors; and

performing the selecting, applying and determining steps through a plurality of iterations each with a different value for "K" and selecting the neural network with a value for "K" and the threshold and weighting values determined which provide the optimal classification.