Automatic model maintenance through local nets

• US 7,444,310 B2
• Filed: 03/28/2003
• Issued: 10/28/2008
• Est. Priority Date: 04/19/2002
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Computer Associates Think Inc. (Broadcom, Inc.)
• Inventors: Duan, Baofu, Pao, Yoh-Han, Meng, Zhuo
• Primary Examiner(s): STARKS, WILBERT L
• Application Number:
  US10/401,930
• Publication Number:
  US 20030200075A1
• Time in Patent Office: 2,041 Days
• Field of Search: 705/15, 702/27, 704/256.5
• US Class Current: 706/15
• CPC Class Codes: G06Q 10/10 Office automation; Time management

Abstract

A model maintenance method is provided. If accuracy of prediction by a current model through consultation with new data is determined to be below a predetermined threshold, a compound model is formed by supplementing the current model with a local net trained with the new data.

Claims

1. A model maintenance method comprising:

collecting new data related to a system model, after the system model is formed;

determining that an accuracy of a prediction by the system model through consultation with the new data is below a predetermined threshold;

forming a compound model by supplementing the system model with a local net trained with the new data; and

storing the compound model on a storage device.

2. The method of claim 1, wherein the local net has an associated valid data space corresponding to a space spanned by the new data.

3. The method of claim 2, wherein when the compound model is consulted with a data point, the local net is consulted with the data point and a result of consulting the local net is returned, if the data point is within the associated valid data space of the local net.

4. The method of claim 3, wherein if the data point is not within the associated valid data space of the local net, the system model is consulted with the data point.

5. The method of claim 1, wherein the compound model is updated by supplementing the original compound model with a second local net formed through training with additional new data, if accuracy of prediction by the compound model through consulting with the additional new data is below the predetermined threshold.

6. The method of claim 5, wherein when the updated compound model is consulted with a data point, the second local net is consulted with the data point and a result of consulting the second local net is returned, if the data point is within a valid data space of the second local net.

7. The method of claim 6, wherein the first local net is consulted with the data point and a result of consulting the first local net is returned, if the data point is not within the valid data space of the second local net and is within the associated valid data space of the first local net.

8. The method of claim 7, wherein the system model is consulted with the data point and a result of consulting the system model is returned, if the data point is not within the valid data space of the second local net and is not within the valid data space of the first local net.

9. The method of claim 1, wherein error of model prediction is determined when the current model is consulted with a new data point, and the new data point is added to a new training set if the error corresponding to consultation of the model with the new data point is not below a data collection threshold.

10. The method of claim 9, wherein the new training set is used to establish a new local net and the current model is updated by supplementing the model with the new local net, if the error corresponding to consultation of the model with the new data point is above a model update threshold.

11. The method of claim 9, wherein when a number of data points in the new training set reaches a maximum number, the new training set is used to establish a new local net, and the current model is updated by supplementing the model with the new local net.

12. The method of claim 9, wherein the new training set is not used to establish a new local net, unless a number of data points in the new training set is equal to or greater than a minimum number.

13. The method of claim 9, wherein outliers are removed from the new training set.

14. The method of claim 1, wherein a clustering technique or decision tree technique is applied to the new data to determine one or more data space ranges associated with the local net.

28. The method of claim 10, further comprising:

upon determining that an accuracy of the predication of the first compound model is below a first threshold, training a second local net, the second local net having a second associated data space;

supplementing the first compound model with the second local net to form a second compound model;

storing the second compound model; and

applying subsequent data to the second compound model, wherein;

upon determining that a data point of the subsequent data is within the second associated data space of the second local net, consulting the second local net with the data point and returning a prediction;

upon determining that a data point of the subsequent data is not within the second associated data space of the second local net and is within the first associated data space of the first local net, consulting the first local net with the data point and returning a prediction; and

upon determining that a data point of the subsequent data is not within the first associated data space of the first local net and is not within the second associated data space of the second local net, consulting the system model with the data point and returning a prediction.

15. A computer system, comprising:

a processor; and

a program storage device readable by the computer system, tangibly embodying a program of instructions executable by the processor to perform a model maintenance method, the method comprising;

collecting new data related to a current model, after the current model is formed;

determining that an accuracy of a prediction by the current model through consultation with the new data is below a predetermined threshold;

forming a compound model by supplementing the current model with a local net trained with the new data; and

storing the compound model.

16. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform a model maintenance method, the method comprising:

collecting new data related to a system model, after the system model is formed;

determining that an accuracy of a prediction by a current model through consultation with the new data is below a predetermined threshold;

forming a compound model by supplementing the current model with a local net trained with the new data; and

storing the compound model.

17. A model maintenance method comprising:

determining that an accuracy of a current model is below a predetermined threshold;

collecting data related to the current model, after the current model is formed, for adaptively updating the current model;

forming a compound model by supplementing the current model with a local net trained with the collected data; and

storing the compound model.

18. The method of claim 17, wherein inadequacy of the prediction accuracy of the current model is attributed to training with training data corresponding to only partial system behavior.

19. The method of claim 17, wherein deterioration of model prediction accuracy is attributed to a shift to system dynamics after the current model was established.

20. The method of claim 19, wherein one or more local nets are added to the compound model to capture the new system dynamics.

21. The method of claim 17, wherein inadequacy of the prediction accuracy of the current model is attributed to a combination of (a) training with training data corresponding to only partial system behavior, and (b) a change of system dynamics after the current model was established.

22. A compound model of a system, comprising:

a current model;

new data related to a current model, the new data collected after the current model is formed;

at least one local net having an associated valid data space,

wherein when the compound model is consulted with a data point of the new data, the local net is consulted with the data point and a result of consulting the local net is returned, if the data point is within an associated valid data space of the local net, and the current model is consulted with the data point and a result of consulting the current model is returned, if the data point is not within the associated valid data space of the local net.

23. The compound model of claim 22, wherein the compound model is updated repeatedly by adding a series of additional local nets formed through training with new data points, if accuracy of predictions by the compound model through consultations with the new data points is below a predetermined threshold.

24. The compound model of claim 23, wherein the updated compound model is consulted with a new data point by comparing the new data point to the valid data spaces of the local nets in reverse order to identify one of the local nets which has a valid data space within which the new data point falls, consulting the identified local net with the new data point, and returning a result of consulting the identified local net with the new data point.

25. The compound model of claim 23, wherein usage of each local net is tracked, and infrequently used local nets are purged when the compound model is updated.

26. A model maintenance method, comprising:

applying data to a system model to return a prediction;

upon determining that the accuracy of the prediction of the system model is below a first threshold, training a first local net, the first local net having a first associated data space;

supplementing the system model with the first local net to form a first compound model;

storing the first compound model; and

applying subsequent data to the first compound model, wherein;

upon determining that a data point of the subsequent data is within the first associated data space of the first local net, consulting the first local net with the data point and returning a prediction; and

upon determining that a data point of the subsequent data is not within the first associated data space of the first local net, consulting the system model with the data point and returning a prediction.

27. The method of claim 26:

further comprising;

collecting and storing at least two data points;

clustering the collected data points into at least one cluster; and

determining the first associated data space based on the collected data points of the at least one cluster; and

wherein training the first local net comprises training the first local net using the collected data points of the at least one cluster.