Automated analyzers for estimation systems

• US 7,127,439 B2
• Filed: 06/07/2002
• Issued: 10/24/2006
• Est. Priority Date: 06/08/2001
• Status: Expired due to Fees

• Current Assignee: Metricly, Inc. (Virtana Corp.)
• Original Assignee: Netuitive, Inc. (Virtana Corp.)
• Inventors: Jannarone, Robert, Rasmussen, Amanda, Homoki, David
• Primary Examiner(s): Knight, Anthony
• Assistant Examiner(s): Brown, Jr., Nathan H.
• Application Number:
  US10/165,232
• Publication Number:
  US 20020188582A1
• Time in Patent Office: 1,600 Days
• Field of Search: 706/26, 702/186
• US Class Current: 706/25
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

This invention specifies analyzers to be run in conjunction with computer estimation systems, which may be applied to performance monitoring (APM) services. A semi-automated analyzer may be used by a human analyst to periodically evaluate available historical data for establishing a desired set of input measurements, model parameters, and reporting criteria, collectively called configuration parameters, to be used by an estimation system. In addition, a fully automated analyzer may periodically and automatically reevaluate such configuration parameters and automatically reconfigure the estimation system accordingly. For both types of analyzer, the active set of input measurements for the computerized estimation system can be initially established or periodically updated to perform any variety of configuration tuning operations, including the following: removing linearly redundant variables; removing inputs showing no variation; removing unnecessary or non-estimable inputs; tuning estimation system operating parameters that govern learning rates and the use of recent trends; occasional model accuracy assessment; and tuning monitoring alarm thresholds.

Claims

1. A method for operating an analyzer for an estimation system operable for receiving input values for successive time trials, computing output values based on the input values and learned parameters, and updating the learned parameters to reflect relationships observed among the input and output values, comprising the steps of:

(a) receiving historical data comprising samples of the input and output values for a plurality of time trials;

(b) receiving a set of model configuration parameters including specifications for a statistical model that may be implemented by the estimation system;

(c) activating the estimation system to run the historical data on the statistical model to compute the output values and the learned parameters for the statistical model;

(d) analyzing the learned parameters to identify input values that are ineffective for estimating the output values;

(e) reducing the size of the model by eliminating the ineffective input values; and

(f) periodically repeating steps (a) though (e) to reinstate into the model previously eliminated input values that have become effective for predicting the output values.

2. The method of claim 1, wherein the step of analyzing the learned parameters to identify input values that are ineffective for estimating the output values comprises basic screening, linear redundancy elimination, and unnecessary input removal.

3. An apparatus configured to perform the method of claim 1 or a computer storage medium storing instructions for causing the apparatus to perform the method of claim 1.

9. The method of claim 1, wherein the configuration parameters identify a number of forecast steps for computing output values.

4. A method for operating an analyzer for an estimation system operable for receiving input values for successive time trials, computing output values based on the input values and learned parameters, and updating the learned parameters to reflect relationships observed among the input and output values, comprising the steps of:

receiving historical data comprising samples of the input and output values for a plurality of time trials;

receiving a plurality of sets of candidate model configuration parameters, each set including specifications for a an intrinsically different statistical model that may be implemented by the estimation system;

activating the estimation system to run the historical data on each statistical model to compute the output values and learned parameters for each statistical model;

performing a qualitative model assessment for each statistical model;

identifying a best-performing statistical model based on the plurality of qualitative model assessments; and

implementing the best-performing statistical model with the estimation system to compute output values for future time trials.

5. An apparatus configured to perform the method of claim 4 or a computer storage medium storing instructions for causing the apparatus to perform the method of claim 4.

6. The method of claim 4, wherein:

the configuration parameters identify a number of historical input values and a power degree for computing a best fit polynomial of the identified degree to the identified historical input values; and

coefficients for the best fit polynomial are used to update the learned parameters.

7. The method of claim 5, wherein the configuration parameters identify a learning parameter for use in updating the learned parameters.

8. The method of claim 7, wherein the plurality of candidate model configuration parameters reflect variations of the parameters comprising the number of historical input value, the polynomial degree, and the learning parameter.

10. A method for operating an analyzer for an estimation system operable for receiving input values for successive time trials, computing output values based on the input values and learned parameters, and updating the learned parameters to reflect relationships observed among the input and output values, comprising the steps of:

receiving historical data comprising samples of the input and output values for a plurality of time trials;

receiving a plurality of sets of candidate model configuration parameters, each set including specifications for an intrinsically different statistical model that may be implemented by the estimation system;

activating the estimation system to run the historical data on each statistical model to compute the output values and learned parameters for each statistical model;

analyzing the learned parameters to identify input values that are ineffective for estimating the output values;

reducing the size of the model by eliminating the ineffective input values;

performing a qualitative model assessment for each statistical model;

selecting a best-performing statistical model for the estimation system based on the plurality of qualitative model assessments; and

implementing the best-performing statistical model with the estimation system to compute output values for future time trials.

11. The method of claim 10, wherein:

the configuration parameters identify a number of historical input values and a power degree for computing a best fit polynomial of the identified degree to the identified historical input values; and

coefficients for the best fit polynomial are used to update the learned parameters.

12. The method of claim 11, wherein the configuration parameters identify a learning parameter for use in updating the learned parameters.

13. The method of claim 12, wherein the plurality of candidate model configuration parameters reflect variations of the parameters comprising the number of historical input value, the power degree, and the learning parameter.

14. The method of claim 13, wherein:

the configuration parameters identify a number of forecast steps for computing output values.

21. An apparatus configured to perform the method of claim 10.

24. A computer storage medium storing instructions for causing said apparatus to perform the method of claim 10.

15. A method for operating an analyzer for an estimation system operable for receiving input values for successive time trials, computing output values based on the input values and learned parameters, and updating the learned parameters to reflect relationships observed among the input and output values, comprising the steps of:

receiving historical data comprising samples of the input and output values for a plurality of time trials;

receiving a set of candidate model configuration parameters including specifications for a statistical model that may be implemented by the estimation system;

activating the estimation system to run the historical data on the statistical model to compute the output values and learned parameters for the statistical model;

computing alert thresholds for output values based on observed deviance values between the computed output values and the historical samples of output values to obtain a desired alert sensitivity;

computing a global deviance value for the output values; and

computing a global deviance threshold for the global deviance value based on observed deviance between the computed output values and the historical samples of output values to obtain a desired alarm sensitivity.

16. An apparatus configured to perform the method of claim 15.

23. A computer storage medium storing instructions for causing said apparatus to perform the method of claim 15.

17. A method for operating an analyzer for an estimation system operable for receiving input values for successive time trials, computing output values based on the input values and learned parameters, and updating the learned parameters to reflect relationships observed among the input and output values, comprising the steps of:

continually running several intrinsically different competing models on the estimation system;

occasionally qualitatively assessing the results from the competing models;

based on the qualitatively assessments;

identifying a best recently performing model; and

generating the output values based on the best recently performing model.

18. The method of claim 17, further operable for:

continually monitoring and adjusting an alarm threshold for the best recently performing model.

19. The method of claim 18, wherein the step of continually monitoring and adjusting an alarm threshold comprises the steps of:

computing a global deviance value for the output values; and

computing a global deviance threshold for the global deviance value based on observed deviance between the computed output values and the historical samples of output values to obtain a desired alarm sensitivity.

20. The method of claim 17, wherein:

the competing models comprise configuration parameters identifying a number of historical input values and a power degree for computing a best fit polynomial of the identified degree to the identified historical input values; and

coefficients for the best fit polynomial are used to update the learned parameters.

22. An apparatus configured to perform the method of claim 17.

25. A computer storage medium storing instructions for causing said apparatus to perform the method of claim 17.