Automated analyzers for estimation systems

• US 20020188582A1
• Filed: 06/07/2002
• Published: 12/12/2002
• Est. Priority Date: 06/08/2001
• Status: Active Grant

• Current Assignee: Metricly, Inc. (Virtana Corp.)
• Original Assignee: Netuitive, Inc. (Virtana Corp.)
• Inventors: Jannarone, Robert, Rasmussen, Amanda, Homoki, David
• Granted Patent: US 7,127,439 B2
• Time in Patent Office: Days
• US Class Current: 706/26
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

This invention specifies analyzers to be run in conjunction with computer estimation systems, which may be applied to performance monitoring (APM) services. A semi-automated analyzer may be used by a human analyst to periodically evaluate available historical data for establishing a desired set of input measurements, model parameters, and reporting criteria, collectively called configuration parameters, to be used by an estimation system. In addition, a fully automated analyzer may periodically and automatically reevaluate such configuration parameters and automatically reconfigure the estimation system accordingly. For both types of analyzer, the active set of input measurements for the computerized estimation system can be initially established or periodically updated to perform any variety of configuration tuning operations, including the following: removing linearly redundant variables; removing inputs showing no variation; removing unnecessary or non-estimable inputs; tuning estimation system operating parameters that govern learning rates and the use of recent trends; occasional model accuracy assessment; and tuning monitoring alarm thresholds.

Claims

1. In or for an estimation system operable for receiving input values for successive time trials and, for each time trial, computing output values based on the input values and learned parameters and updating the learned parameters to reflect relationships observed among the input and output values, an analyzer operable for performing the steps of:

(a) receiving historical data comprising samples of the input and output values for a plurality of time trials;

(b) receiving a set of model reduction configuration parameters including specifications for a statistical model that may be implemented by the estimation system;

(c) activating the estimation system to run the historical data on the statistical model to compute the output values and the learned parameters for the statistical model;

(d) analyzing the learned parameters to identify input values that are ineffective for estimating the output values;

(e) reducing the size of the model by eliminating the ineffective input values.

2. The analyzer of claim 1, further operable for:

(f) repeating steps (b) though (e) to perform a series of model reduction steps comprising basic screening, linear redundancy elimination, and unnecessary input removal.

3. The analyzer of claim 2, further operable for periodically repeating steps (a) though (f) to reinstate into the model previously eliminated input values that have become effective for predicting the output values.

9. The analyzer of claim 1, wherein the configuration parameters identify a number of forecast steps for computing output values.

4. In or for an estimation system operable for receiving input values for successive time trials and, for each time trial, computing output values based on the input values and learned parameters and updating the learned parameters to reflect relationships observed among the input and output values, an analyzer operable for performing the steps of:

(a) receiving historical data comprising samples of the input and output values for a plurality of time trials;

(b) receiving a set of candidate model configuration parameters including specifications for a statistical model that may be implemented by the estimation system;

(c) activating the estimation system to run the historical data on the statistical model to compute the output values and learned parameters for the statistical model; and

(d) performing a model assessment by comparing the computed output values to the historical samples of output values.

5. The analyzer of claim 4, further operable for:

repeating steps (b) through (d) for a plurality of candidate model configuration parameters; and

identifying a desired set of the configuration parameters based on the plurality of model assessments.

6. The analyzer of claim 4, wherein:

the configuration parameters identify a number of historical input values and a power degree for computing a best fit polynomial of the identified degree to the identified historical input values; and

coefficients for the best fit polynomial are used to update the learned parameters.

7. The analyzer of claim 5, wherein the configuration parameters identify a learning parameter for use in updating the learned parameters.

8. The analyzer of claim 7, wherein the plurality of candidate model configuration parameters reflect variations of the parameters comprising the number of historical input value, the polynomial degree, and the learning parameter.

10. In or for an estimation system operable for receiving input values for successive time trials and, for each time trial, computing output values based on the input values and learned parameters and updating the learned parameters to reflect relationships observed among the input and output values, an analyzer operable for performing the steps of:

(a) receiving historical data comprising samples of the input and output values for a plurality of time trials;

(b) receiving a set of model reduction configuration parameters including specifications for a statistical model that may be implemented by the estimation system;

(c) activating the estimation system to run the historical data on the statistical model to compute the output values and the learned parameters for the statistical model;

(d) analyzing the learned parameters to identify input values that are ineffective for estimating the output values;

(e) reducing the size of the model by eliminating the ineffective input values;

(f) repeating steps (b) though (e) to perform a series of model reduction steps comprising basic screening, linear redundancy elimination, and unnecessary input removal to create a reduced size statistical mode;

(g) receiving a set of candidate model configuration parameters including specifications for the reduced size statistical model that may be implemented by the estimation system;

(h) activating the estimation system to run the historical data on the reduced size statistical model to compute the output values and learned parameters for the statistical model; and

(i) performing a model assessment by comparing the computed output values to the historical samples of output values;

(j) repeating steps (g) through (i) for a plurality of candidate model configuration parameters;

(k) identifying a desired set of the configuration parameters based on the plurality of model assessments.

11. The analyzer of claim 10, wherein:

the configuration parameters identify a number of historical input values and a power degree for computing a best fit polynomial of the identified degree to the identified historical input values; and

coefficients for the best fit polynomial are used to update the learned parameters.

12. The analyzer of claim 11, wherein the configuration parameters identify a learning parameter for use in updating the learned parameters.

13. The analyzer of claim 12, wherein the plurality of candidate model configuration parameters reflect variations of the parameters comprising the number of historical input value, the power degree, and the learning parameter.

14. The analyzer of claim 13, wherein:

the configuration parameters identify a number of forecast steps for computing output values.

16. The analyzer of claim 15, further operable for:

computing a global deviance value for the output values; and

computing a global deviance threshold for the global deviance value based on observed deviance between the computed output values and the historical samples of output values to obtain a desired alarm sensitivity.

18. The analyzer of claim 17, further operable for:

continually monitoring and adjusting an alarm threshold for the best recently performing model.

19. The analyzer of claim 18, wherein the step of continually monitoring and adjusting an alarm threshold comprises the steps of:

computing a global deviance value for the output values; and

computing a global deviance threshold for the global deviance value based on observed deviance between the computed output values and the historical samples of output values to obtain a desired alarm sensitivity.

20. The analyzer of claim 17, wherein:

the competing models comprise configuration parameters identifying a number of historical input values and a power degree for computing a best fit polynomial of the identified degree to the identified historical input values; and

coefficients for the best fit polynomial are used to update the learned parameters.

15. In or for an estimation system operable for receiving input values for successive time trials and, for each time trial, computing output values based on the input values and learned parameters and updating the learned parameters to reflect relationships observed among the input and output values, an analyzer operable for performing the steps of:

receiving historical data comprising samples of the input and output values for a plurality of time trials;

receiving a set of candidate model configuration parameters including specifications for a statistical model that may be implemented by the estimation system;

activating the estimation system to run the historical data on the statistical model to compute the output values and learned parameters for the statistical model; and

computing alert thresholds for output values based on observed deviance values between the computed output values and the historical samples of output values to obtain a desired alert sensitivity.

17. In or for an estimation system operable for receiving input values for successive time trials and, for each time trial, computing output values based on the input values and learned parameters and updating the learned parameters to reflect relationships observed among the input and output values, an analyzer operable for performing the steps of:

continually running several competing models of the estimation system;

occasionally comparing results from the competing models;

based on the comparison;

identifying a best recently performing model; and

generating the output values based on the best recently performing model.