Method and apparatus for operating a neural network with missing and/or incomplete data

• US 6,591,254 B1
• Filed: 11/06/2001
• Issued: 07/08/2003
• Est. Priority Date: 09/20/1995
• Status: Expired due to Fees

• Current Assignee: Rockwell Automation Technologies Incorporated (Allen-Bradley Co., Inc.)
• Original Assignee: Pavilion Technologies Incorporated (Rockwell Automation, Inc.)
• Inventors: Ferguson, Ralph Bruce, Hartman, Eric Jon, Keeler, James David
• Primary Examiner(s): Shah, Sanjiv
• Application Number:
  US10/040,085
• Time in Patent Office: 609 Days
• Field of Search: 706/21, 706/22, 706/25, 706/15, 706/23, 706/26
• US Class Current: 706/21
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs

Abstract

A neural network system is provided that models the system in a system model (12) with the output thereof providing a predicted output. This predicted output is modified or controlled by an output control (14). Input data is processed in a data preprocess step (10) to reconcile the data for input to the system model (12). Additionally, the error resulted from the reconciliation is input to an uncertainty model to predict the uncertainty in the predicted output. This is input to a decision processor (20) which is utilized to control the output control (14). The output control (14) is controlled to either vary the predicted output or to inhibit the predicted output whenever the output of the uncertainty model (18) exceeds a predetermined decision threshold, input by a decision threshold block (22). Additionally, a validity model (16) is also provided which represents the reliability or validity of the output as a function of the number of data points in a given data region during training of the system model (12). This predicts the confidence in the predicted output which is also input to the decision processor (20). The decision processor (20) therefore bases its decision on the predicted confidence and the predicted uncertainty. Additionally, the uncertainty output by the data preprocess block (10) can be utilized to train the system model (12).

Claims

1. A network for estimating the error in the prediction output space of a predictive system model operating over a prediction input space, comprising:

an input for receiving an input vector comprising a plurality of input values that occupy the prediction input space;

an output for outputting an output prediction error vector that occupies an output space corresponding to the prediction output space of the predictive system model; and

a processing layer for mapping the prediction input space to the prediction output space through a representation of the prediction error in the predictive system model to provide said output prediction error vector.

2. The network of claim 1, and further comprising:

a preprocess input for receiving an unprocessed data input vector having associated therewith unprocessed data associated with substantially the same input space as said input vector, said unprocessed data input vector having errors associated with the associated unprocessed data in select portions of the prediction input space; and

a data preprocessor for processing the unprocessed data in the unprocessed data input vector to minimize the errors therein to provide said input vector on an output.

3. The network of claim 2, wherein said unprocessed data input vector is comprised of data having portions thereof that are unusable and said data preprocessor comprises a reconciliation device for reconciling the unprocessed data to replace the unusable portions with reconciled data.

4. The network of claim 2, wherein said data prepossessor is operable to calculate and output the uncertainty for each value output by said data preprocessor.

5. The network of claim 1, wherein the predictive system model comprises a non-linear model having an input for receiving the input vector that is within the prediction input space and an output for outputting a predicted output vector within the prediction output space, said non-linear model mapping the prediction input space to the prediction output space through a non-linear representation of a system.

6. The network of claim 5, wherein the predictive system model is trained on a set of training data having uncertainties associated therewith and wherein said processing layer is operable to map the prediction input space to the prediction output space through a representation of the combined prediction error in the predictive system model and the prediction error in the set of training due to the uncertainty in the set of training data.

7. The network of claim 5 and further comprising:

a plurality of decision thresholds for defining predetermined threshold values for said output prediction error vector;

an output control for effecting a change in the value of said predicted output vector from the predictive system model; and

a decision processor for receiving said output prediction error vector and comparing it to said decision thresholds and operating said output control to effect said change on the value of said predicted output vector when the value of said output prediction error vector meets a predetermined relationship with respect to said decision thresholds.

8. The network of claim 6, wherein said non-linear representation is a trained representation that is trained on a finite set of input data within the input space in accordance with a predetermined training algorithm and further comprising a validity model for providing a representation of the validity of the predicted output vector of the system model for a given value of the input vector within the input space, said validity model having:

an input for receiving the input vector within the input space;

an output for outputting a validity output vector corresponding to the output space;

a validity processor for generating said validity output vector in response to input of said input vector and the location of said input vector in the input space, the value of said validity output vector corresponding to the amount of training data on which the system model was trained in the region of the input space about the value of the input vector.

9. The network of claim 8, and further comprising:

a plurality of decision thresholds for defining predetermined threshold values for the validity output vector;

an output control for effecting a change in the value of said predicted output vector from the predictive system model; and

a decision processor for receiving said validity output vector and comparing said validity output vector to said decision thresholds, and operating said output control to effect said change in the value of said predicted output vector when the value of said validity output vector meets a predetermined relationship with respect to said decision thresholds.

10. A network for providing a measure of the validity in the prediction output space of a predictive system model that provides a prediction output and operates over a prediction input space, comprising:

an input for receiving an input vector comprising a plurality of input values that occupy the prediction input space;

an output for outputting a validity measure output vector that occupies an output space corresponding to the prediction output space of the predictive system model; and

a processing layer for mapping the prediction input space to the prediction output space through a representation of the validity of the system model that was learned on a set of training data, the representation of the validity of the system model being a function of the distribution of the training data in the prediction input space that was input thereto during training to provide a measure of the validity of the system model prediction output.

11. The network of claim 10, and further comprising:

a preprocess input for receiving an unprocessed data input vector having associated therewith unprocessed data associated with substantially the same input space as said input vector, said unprocessed data input vector having errors associated with the associated unprocessed data in select portions of the prediction input space; and

a data preprocessor for processing the unprocessed data in the unprocessed data input vector to minimize the errors therein to provide said input vector on an output.

12. The network of claim 11, wherein said unprocessed data input vector is comprised of data having portions thereof that are unusable and said data preprocessor comprises a reconciliation device for reconciling data to replace the unusable portions with reconciled data.

13. The network of claim 12, wherein said data preprocessor is operable to calculate and output the uncertainty for each value of reconciled data output by said data preprocessor.

14. The network of claim 10, wherein the predictive system model comprises a non-linear model having an input for receiving the input vector that is within the prediction input space and an output for outputting a predicted output vector within the prediction output space, said non-linear model mapping the prediction input space to the prediction output space through a non-linear representation of a system.

15. A network for providing a measure of the validity in the prediction output space of a predictive system model that is comprised of a non-linear model that provides a prediction output and operates over a prediction input space, comprising:

an input for receiving an input vector comprising a plurality of input values that occupy the prediction input space;

an output for outputting a validity measure output vector that occupies an output space corresponding to the prediction output space of the predictive system model;

a processing layer for mapping the prediction input space to the prediction output space through a non-linear representation of the validity of the system model that was learned on a set of training data, the representation of the validity of the system model being a function of the distribution of the training data in the prediction input space that was input thereto during training to provide a measure of the validity of the system model prediction output;

a plurality of decision thresholds for defining predetermined threshold values for said validity measure output vector;

an output control for effecting a change in the value of said predicted output vector from the predictive system model; and

a decision processor for receiving said validity measure output vector and comparing it to said decision threshold and operating said output control to effect said change on the value of said predicted output vector when the value of said validity measure output vector meets a predetermined relationship with respect to said decision threshold.

16. A network for providing a measure of the validity in the prediction output space of a predictive system model that provides a prediction output and operates over a prediction input space, comprising:

an input for receiving an input vector comprising a plurality of input values that occupy the prediction input space;

an output for outputting a validity measure output vector that occupies an output space corresponding to the prediction output space of the predictive system model; and

a processing layer for mapping the prediction input space to the prediction output space through a representation of the validity of the system model that was learned on a set of training data, the representation of the validity of the system model being a function of the distribution of the training data in the prediction input space that was input thereto during training to provide a measure of the validity of the system model prediction output;

wherein said processing layer comprises;

a memory for storing a profile of the training data density over the input space, and

a processor for processing the location of the input data in the input space and the density of the training data at said location as defined by said stored profile to generate said validity measure output vector as a function of the distribution of said training data proximate to the location in the input space of the input data.

17. A method for estimating the error in the prediction output space of a predictive system model over a prediction input space, comprising the steps of:

receiving an input vector comprising a plurality of input values that occupy the prediction input space;

outputting an output prediction error vector that occupies an output space corresponding to the prediction output space of the predictive system model; and

mapping the prediction input space to the prediction output space through a representation of the prediction error in the predictive system model to provide the output prediction error vector in the step of outputting.

18. The method of claim 17, and further comprising the steps of:

receiving an unprocessed data input vector having associated therewith unprocessed data associated with substantially the same input space as the input vector, the unprocessed data input vector having errors associated with the associated unprocessed data in select portions of the prediction input space; and

processing the unprocesssed data in the unprocessed data vector to minimize the errors therein to provide the input vector on an output.

19. The method of claim 18, wherein the step of receiving an unprocessed data input vector comprises receiving an unprocessed data input vector that is comprised of data having portions thereof that are unusable and the step of processing the unprocessed data comprises reconciling the unprocessed data to replace the unusable portions with reconciled data.

20. The method of claim 19, wherein the step of processing the data is further operable to calculate and output the uncertainty for each value of the reconciled data output by the step of processing.