Process model monitoring method and system

• US 20060230097A1
• Filed: 04/08/2005
• Published: 10/12/2006
• Est. Priority Date: 04/08/2005
• Status: Abandoned Application

• Current Assignee: Caterpillar Incorporated
• Original Assignee: Caterpillar Incorporated
• Inventors: Grichnik, Anthony J., Seskin, Michael
• Application Number:
  US11/101,531
• Publication Number:
  US 20060230097A1
• Time in Patent Office: Days
• US Class Current: 708/803
• CPC Class Codes: G05B 17/02 electric

Abstract

A computer-implemented method is provided for monitoring model performance. The method may include obtaining configuration information and obtaining operational information about a computational model and a system being modeled. The computational model and the system may include a plurality of input parameters and one or more output parameters. The system may generate respective actual values of the one or more output parameters, and the computational model may predict respective values of the one or more output parameters. The method may also include applying an evaluation rule from a rule set, based on the configuration information, to the operational information to determine whether the rule is satisfied.

Claims

1. A computer-implemented method for monitoring model performance, comprising:

obtaining configuration information;

obtaining operational information about a computational model and a system being modeled, wherein the computational model and the system include a plurality of input parameters and one or more output parameters, the system generates respective actual values of the one or more output parameters, and the computational model predicts respective values of the one or more output parameters; and

applying an evaluation rule from a rule set, based on the configuration information, to the operational information to determine whether the rule is satisfied.

2. The method according to claim 1, further including:

sending out a trigger if the evaluation rule is satisfied to indicate a decrease in a performance of the computational model.

3. The method according to claim 1, wherein the configuration information includes an enable or disable command for enabling or disabling the monitoring.

4. The method according to claim 1, wherein the computational model is created by:

obtaining data records associated with one or more input variables and the one or more output parameters;

selecting the plurality of input parameters from the one or more input variables;

generating the computational model indicative of interrelationships between the plurality of input parameters and the one or more output parameters based on the data records; and

determining desired respective statistical distributions of the plurality of input parameters of the computational model.

5. The method according to claim 4, wherein selecting further includes:

pre-processing the data records; and

using a genetic algorithm to select the plurality of input parameters from the one or more input variables based on a mahalanobis distance between a normal data set and an abnormal data set of the data records.

6. The method according to claim 4, wherein generating further includes:

creating a neural network computational model;

training the neural network computational model using the data records; and

validating the neural network computation model using the data records.

7. The method according to claim 4, wherein determining further includes:

determining a candidate set of input parameters with a maximum zeta statistic using a genetic algorithm; and

determining the desired distributions of the input parameters based on the candidate set,

wherein the zeta statistic ζ

is represented by;

$\mathrm{\zeta <br><br>}=\stackrel{j}{\underset{1}{\sum <br><br>}}<br><br>\stackrel{i}{\underset{1}{\sum <br><br>}}<br><br><br><br>S_{\mathrm{ij}}<br><br><br><br>\left(\frac{{\mathrm{\sigma <br><br>}}_i}{{\stackrel{\_}{x}}_i}\right)<br><br>\left(\frac{{\stackrel{\_}{x}}_j}{{\mathrm{\sigma <br><br>}}_j}\right),$

provided that {overscore (x)}_i represents a mean of an ith input;

{overscore (x)}_j represents a mean of a jth output;

σ

_i represents a standard deviation of the ith input;

σ

_j represents a standard deviation of the jth output; and

|S_ij| represents sensitivity of the jth output to the ith input of the computational model.

8. The method according to claim 1, wherein applying includes:

determining a divergence between the predicted values of the one or more output parameters from the computational model and the actual values of the one or more output parameters from the system;

determining whether the divergence is beyond a predetermined threshold; and

determining that a decreased performance condition of the computational model exists if the divergence is beyond the threshold.

9. The method according to claim 1, wherein applying includes:

determining a divergence between the predicted values of the one or more output parameters from the computational model and the actual values of the one or more output parameters from the system;

determining whether the divergence is beyond a predetermined threshold;

recording a number of occurrences of the divergence being beyond the predetermined threshold; and

determining that a decreased performance condition of the computational model exists if the number of occurrences of the divergence is beyond a predetermined number.

10. The method according to claim 1, wherein applying includes:

determining a time period for the computational model;

determining whether the time period is beyond a predetermined threshold; and

determining whether an expiration condition of the computational model exists if the time period is beyond the threshold.

11. The method according to claim 1, wherein the operational information includes at least:

the actual values of the one or more output parameters,

the predicted values of the one or more output parameters; and

a usage history including a time period during which the computational model is not used.

12. A computer system, comprising:

a database configured to store data records associated with a computational model, a plurality of input parameters, and one or more output parameters; and

a processor configured to;

obtain configuration information;

obtain operational information about the computational model from the database, wherein the computational model and a system being modeled include the plurality of input parameters and the one or more output parameters, the system generates respective actual values of the one or more output parameters, and the computational model predicts respective values of the one or more output parameters; and

apply an evaluation rule from a rule set, based on the configuration information, to the operational information to determine whether the evaluation rule is satisfied.

13. The computer system according to claim 12, wherein the processor is further configured to:

send out a trigger if the evaluation rule is satisfied to indicate a decrease in a performance of the computational model.

14. The computer system according to claim 12, wherein the computational model is created by:

obtaining data records associated with one or more input variables and the one or more output parameters;

selecting the plurality of input parameters from the one or more input variables;

generating the computational model indicative of interrelationships between the plurality input parameters and the one or more output parameters based on the data records; and

determining desired respective statistical distributions of the plurality of input parameters of the computational model.

15. The computer system according to claim 14, wherein selecting further includes:

pre-processing the data records; and

using a genetic algorithm to select the plurality of input parameters from one or more input variables based on a mahalanobis distance between a normal data set and an abnormal data set of the data records.

16. The computer system according to claim 14, wherein determining further includes:

determining a candidate set of input parameters with a maximum zeta statistic using a genetic algorithm; and

determining the desired statistical distributions of the input parameters based on the candidate set,

wherein the zeta statistic ζ

is represented by;

$\mathrm{\zeta <br><br>}=\stackrel{j}{\underset{1}{\sum <br><br>}}<br><br>\stackrel{i}{\underset{1}{\sum <br><br>}}<br><br><br><br>S_{\mathrm{ij}}<br><br><br><br>\left(\frac{{\mathrm{\sigma <br><br>}}_i}{{\stackrel{\_}{x}}_i}\right)<br><br>\left(\frac{{\stackrel{\_}{x}}_j}{{\mathrm{\sigma <br><br>}}_j}\right),$

provided that {overscore (x)}_i represents a mean of an ith input;

{overscore (x)}_j represents a mean of a jth output;

σ

_i represents a standard deviation of the ith input;

σ

_j represents a standard deviation of the jth output; and

|S_ij| represents sensitivity of the jth output to the ith input of the computational model.

17. The computer system according to claim 12, wherein, to apply the evaluation rule, the processor is further configured to:

determine a divergence between the predicted values of the one or more output parameters from the computational model and the actual values of the one or more output parameters from the system;

determine whether the divergence is beyond a predetermined threshold; and

determine that a decreased performance condition of the computational model exists if the divergence is beyond the threshold.

18. The computer system according to claim 12, wherein, to apply the evaluation rule, the processor is further configured to:

determine a time period during which the computational model has not been used;

determine whether the time period is beyond a predetermined threshold; and

determine whether an expiration condition of the computational model exists if the time period is beyond the threshold.

19. A computer-readable medium for use on a computer system configured to perform a model monitoring procedure, the computer-readable medium having computer-executable instructions for performing a method comprising:

obtaining configuration information;

obtaining operational information about a computational model and a system being modeled, wherein the computational model and the system include a plurality of input parameters and one or more output parameters, the system generates respective actual values of the one or more output parameters, and the computational model predicts respective values of the one or more output parameters; and

applying an evaluation rule from a rule set, based on the configuration information, to the operational information to determine whether the evaluation rule is satisfied.

20. The computer-readable medium according to claim 19, wherein the method further includes:

sending out a trigger to indicate a decrease in a performance of the computational model if the evaluation rule is satisfied.

21. The computer-readable medium according to claim 19, wherein applying further includes:

determining a divergence between the predicted values of the one or more output parameters from the computational model and the actual values of the one or more output parameters from the system;

determining whether the divergence is beyond a predetermined threshold; and

determining that a decreased performance condition of the computational model exists if the divergence is beyond the threshold.

22. The computer-readable medium according to claim 19, wherein applying further includes:

determining a time period during which the computational model has not been used;

determining whether the time period is beyond a predetermined threshold; and

determining whether an expiration condition of the computational model exists if the time period is beyond the threshold.

23. The computer-readable medium according to claim 19, wherein the operational information includes at least:

the actual values of the one or more output parameters;

the predicted values of the one or more output parameters; and

a usage history including a time period during which the computational model is not used.