Computer-implemented systems and methods for updating predictive models

• US 8,214,308 B2
• Filed: 05/13/2008
• Issued: 07/03/2012
• Est. Priority Date: 10/23/2007
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Chu, Chengwen Robert
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Hill, Stanley K
• Application Number:
  US12/119,624
• Publication Number:
  US 20090106178A1
• Time in Patent Office: 1,512 Days
• Field of Search: 706/12, 706/14
• US Class Current: 706/14
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

Computer-implemented systems and methods for updating champion predictive models that operate within a production environment. A system and method can include evaluating the predictive model performance of a champion predictive model. Based upon an indication of decay of predictive model performance of the champion predictive model, a corrective action is performed to correct the performance of the champion predictive model.

Claims

1. A computer-implemented method of assessing a predictive model, comprising:

accessing, using one or more data processors, a predictive model for scoring transactional data and data associated with a time period;

generating, using the one or more data processors, one or more predictions using the predictive model and the data;

calculating, using the one or more data processors, a baseline performance metric for the predictive model based upon the predictions;

accessing, using the one or more data processors, new data associated with a new time period that is later than the time period;

generating, using the one or more data processors, one or more new data predictions using the predictive model and the new data;

calculating, using the one or more data processors, an updated performance metric for the predictive model based upon the new data predictions;

comparing, using the one or more data processors, the baseline performance metric and the updated performance metric to determine an indication of predictive ability decay for the predictive model; and

performing, using the one or more data processors, a corrective action for the predictive model when the indication of predictive ability decay reaches a threshold.

2. The method of claim 1, wherein the predictive model is operating within a production environment, wherein performance of an additional predictive model in the production environment is assessed, and wherein a second corrective action is performed on the additional predictive model when predictive ability decay of the additional predictive model reaches the threshold.

3. The method of claim 1, wherein the predictive model generates predictions for fraud detection, rate making, credit scoring, customer retention, customer lifetime value, customer attrition or churn, or marketing response.

4. The method of claim 1, wherein the predictive ability decay is caused by evolution of a customer base or changes in predictability of data used by the predictive model.

5. The method of claim 1, wherein the corrective action is performed by one or more servers or a single general purpose computer.

6. The method of claim 1, wherein the predictive model is a neural network model, a logistic model, or a decision tree model.

7. The method of claim 1, wherein the corrective action includes providing a warning via a graphical interface.

8. The method of claim 7, further comprising:

performing a second corrective action for the predictive model when the indication of the predictive ability decay reaches a second threshold, wherein the second corrective action includes issuing an alert via the graphical interface.

9. The method of claim 8, wherein the threshold is indicative of less severe predictive decay than the second threshold.

10. The method of claim 1, wherein comparing further includes identifying a change in an input or output variable distribution, and wherein the identified change is greater than a distribution change threshold.

11. The method of claim 1, wherein the corrective action includes building a replacement predictive model.

12. The method of claim 11, wherein the replacement predictive model is automatically recalibrated using the new data.

13. The method of claim 1, wherein the corrective action includes sending a notification to pre-selected parties.

14. The method of claim 1, further comprising:

logging the baseline performance metric and the updated performance metric in a data store.

15. The method of claim 14, further comprising:

providing monitoring reports showing trends in model performance.

16. The method of claim 1, wherein the predictive model is retrieved from a centralized model repository.

17. The method of claim 1, wherein the indication of predictive ability decay is calculated according to a Lift5decay, wherein the Lift5decay identifies a lift performance decay based upon top 5% of the data and the new data, wherein the Lift5decay is calculated according to:

Lift5decay=(Lift5_B−

Lift5_A)/Lift5_A,

where Lift5_A is the lift value of the predictions of the predictive model for the top 5% based on the data, where Lift5_B is the lift value of the new predictions of the predictive model for the top 5% based on the new data.

18. The method of claim 1, wherein the indication of predictive ability decay is calculated according to a Lift10decay, wherein the Lift10decay identifies a lift performance decay based on a top 10% of the data and the new data, wherein the Lift10decay is calculated according to:

Lift10decay=(Lift10_B−

Lift10_A)/Lift10_A,

where Lift10_A is the lift value of the predictions of the predictive model for the top 10% based on the data, where Lift10_B is the lift value of the new predictions of the predictive model for the top 10% based on the new data.

19. The method of claim 1, wherein the indication of predictive ability decay is calculated according to a GiniDecay, wherein the GiniDecay identifies a decline in an area under a receiver operating curve, wherein the GiniDecay is calculated according to:

GiniDecay=(GiniIndex_B−

GiniIndex_A)/GiniIndex_A,

where GiniIndex_A is the area under the receiver operating curve for the predictions of the predictive model based on the data, where GiniIndex_B is the area under the receiver operating curve for the new predictions of the predictive model based on the new data.

20. The method of claim 1, wherein the indication of predictive ability decay is calculated according to a KSDecay, wherein the KSDecay identifies a decline in a maximum vertical distance between two curves of a Kolmogorov-Smirnov chart, wherein the KSDecay is calculated according to:

KSDecay=(KSStatistic_B−

KSStatistic_A)/KSStatistic_A,

where KSStatistic_A is the vertical distance between the two curves for the predictions of the predictive model based on the data, where KSStatistic_B is the vertical distance between the two curves for the new predictions of the predictive model based on the new data.

21. The method of claim 17, wherein the data and the new data include actual values.

22. A system for assessing a predictive model, comprising:

one or more data processors;

one or more computer-readable storage mediums containing instructions configured to cause the one or more processors to perform operations including;

accessing a predictive model for scoring transactional data and data associated with a time period;

generating one or more predictions using the predictive model and the data;

calculating a baseline performance metric for the predictive model based upon the predictions;

accessing new data associated with a new time period that is later than the time period;

generating one or more new data predictions using the predictive model and the new data;

calculating an updated performance metric for the predictive model based upon the new data predictions;

comparing the baseline performance metric and the updated performance metric to determine an indication of predictive ability decay for the predictive model; and

performing a corrective action for the predictive model when the indication of predictive ability decay reaches a threshold.

23. A computer-program product for assessing a predictive model, tangibly embodied in a machine-readable storage medium, including instructions configured to cause a data processing apparatus to:

access a predictive model for scoring transactional data and data associated with a time period;

generate one or more predictions using the predictive model and the data;

calculate a baseline performance metric for the predictive model based upon the predictions;

access new data associated with a new time period that is later than the time period;

generate one or more new data predictions using the predictive model and the new data;

calculate an updated performance metric for the predictive model based upon the new data predictions;

compare the baseline performance metric and the updated performance metric to determine an indication of predictive ability decay for the predictive model; and

perform a corrective action for the predictive model when the indication of predictive ability decay reaches a threshold.