Predictive model validation

• US 20050234753A1
• Filed: 04/16/2004
• Published: 10/20/2005
• Est. Priority Date: 04/16/2004
• Status: Active Grant

• Current Assignee: Longhorn Automotive Group LLC (Alpha Alpha Intellectual Partners LLC)
• Original Assignee: Brindle Data LLC (Intellectual Ventures LLC)
• Inventors: Pinto, Stephen K., Jacobs, Marc, Mansfield, Richard, Rubin, Donald
• Granted Patent: US 8,170,841 B2
• Time in Patent Office: Days
• US Class Current: 705/7
• CPC Class Codes: G06Q 10/04 Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" market predictions or forecasting for commercial activities G06Q30/0202
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 40/03 Credit; Loans; Processing thereof

Abstract

Models are generated using a variety of tools and features of a model generation platform. For example, in connection with a project in which a user generates a predictive model based on historical data about a system being modeled, the user is provided through a graphical user interface a structured sequence of model generation activities to be followed, the sequence including dimension reduction, model generation, model process validation, and model re-generation. In connection with a project in which a user generates a predictive model based on historical data about a system being modeled, the user is enabled to validate the model development process with cross-validation between at least two subsets of the historical data; the validated model development process is enabled to be reapplied.

Claims

1. A machine-base method comprising

in connection with a project in which a user generates a predictive model based on historical data about a system being modeled, enabling the user to validate a model process with a predictive model between at least two subsets of the historical data.

2. The method of claim 1 in which the user interface display project goals enabling the user to assess model project performance wherein the project goals comprise at least one of:

cumulative lift over the interval of interest, degree of monotonicity, minimum false negatives or minimum false positives as appropriate.

3. The method of claim 2 also including

identifying that the model does not produce at least a predefined degree of lift for at least one of the validation datasets.

4. The method of claim 3 also including

enabling a user to choose interactively at least one model development criterion change or transformation or interaction of variables to improve the fit of the model.

5. The method of claim 4 also including

graphically displaying and comparing measures of performance for the validation dataset and the training dataset.

6. A machine-based method comprising

in connection with a process in which a user generates a predictive model based on historical data about a system being modeled using the validated model development process, enabling automatic transformations of variables of the data, automatic generation of a predictive model, and automatically generating performance measures of the model on at least two independent datasets of historical data.

7. The method of claim 6 also including

generating measures of the performance of the model for the two datasets, the performance measures being generated separately percentile by percentile.

8. The method of claim 6 also including

graphically displaying and comparing measures of performance for the two datasets.

9. The method of claim 6 also including

persistently storing the validated model development process and the validated model for computing propensities for at least one target outcome variable, the propensities serving as indices of the score for non-historical data.

10. The method of claim 6 also including providing a user interface for assessing project goals against performance.

11. The method of claim 6 also including providing a user interface for selecting at least one subset of the historical data in addition to the training subset.

12. The method of claim 6 providing a user interface for displaying the performance of the model for at least two subsets of the historical data for the interval of interest.

13. The method of claim 6 enabling a user to choose interactively at least one transformation or interaction of variables to improve the model validation process.

14. The method of claim 6 determining whether the model generalizes to the data other than the subsample, and, if so, applying the possible model to all of the data to generate a final model, and cross-validating the final model using random portions of the data.

15. The method of claim 6 providing a user interface that enables the user to select at least one validation dataset and invoke a model process validation method.

16. The method of claim 6 providing a user interface that enables the user to point and click to cause display of information about the model process validation.

17. The method of claim 16 in which the information about the model process validation includes at least one of:

a statistical report card with a link to the statistical report chart, a cumulative lift chart with a link to the cumulative lift chart, and a non-cumulative lift chart with a link to the non-cumulative lift chart.

18. The method of claim 17 in which invocation of the link to the statistical report card causes display of the statistics of model process validation.

19. The method of claim 17 in which invocation of the link to the cumulative lift chart causes display of a cumulative lift chart.

20. The method of claim 17 in which invocation of the link to the cumulative lift chart causes display of a non-cumulative lift chart.

21. The method of claim 17 in which a user is enabled to choose interactively at least one performance criterion change or transformation or interaction of variables to improve the model validation process.

22. The method of claim 6 also including providing a user interface that enables the user to select at least one machine automated model development process applied to the entire dataset for a validated model process.

23. The method of claim 6 also including providing a user interface that enables the user to point and click to cause display of information about the performance of the validated model process applied to the entire set of historical data.

24. The method of claim 23 in which the information about the model performance for two independent data subsets includes at least one of the following:

a statistical report card with a link to the statistical report chart, a cumulative lift chart with a link to the cumulative lift chart, a non-cumulative lift chart with a link to the non-cumulative lift chart.

25. The method of claim 24 in which the invocation of the link to the statistical report card causes display of the statistics of model process validation.

26. The method of claim 24 in which the invocation of the link to the cumulative lift chart causes display of a cumulative lift chart.

27. The method of claim 24 in which the invocation of the link to the cumulative lift chart causes display of a non-cumulative lift chart.

28. The method of claim 6 also including storing the final model and the model process validation results persistently.