Systems and techniques for determining the predictive value of a feature

• US 10,366,346 B2
• Filed: 10/21/2016
• Issued: 07/30/2019
• Est. Priority Date: 05/23/2014
• Status: Active Grant

• Current Assignee: DataRobot, Inc.
• Original Assignee: DataRobot, Inc.
• Inventors: Achin, Jeremy, DeGodoy, Thomas, Conort, Xavier, Yurgenson, Sergey, Steadman, Mark L., Koundry, Glen
• Primary Examiner(s): Sitiriche, Luis A
• Application Number:
  US15/331,797
• Publication Number:
  US 20180060738A1
• Time in Patent Office: 1,012 Days
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 10/04 Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" market predictions or forecasting for commercial activities G06Q30/0202
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/067 Enterprise or organisation modelling
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data

Abstract

A method for determining the predictive value of a feature may include: (a) performing predictive modeling procedures associated with respective predictive models, wherein performing each modeling procedure includes fitting the associated model to an initial dataset representing an initial prediction problem; (b) determining a first accuracy score of each of the fitted models, representing an accuracy with which the fitted model predicts an outcome of the initial prediction problem; (c) shuffling values of a feature across observations included in the initial dataset, thereby generating a modified dataset representing a modified prediction problem; (d) determining a second accuracy score of each of the fitted models, representing an accuracy with which the fitted model predicts an outcome of the modified prediction problem; and (e) determining a model-specific predictive value of the feature for each of the fitted models based on the first and second accuracy scores of the fitted model.

Claims

1. A computer-implemented method for building a predictive model, comprising:

determining a multi-model predictive value of a feature of an initial dataset representing a prediction problem, wherein the initial dataset includes a plurality of observations and each observation includes respective values for a plurality of features, including;

(a) performing one or more predictive modeling procedures, wherein each of the predictive modeling procedures is associated with a different type of predictive model, wherein performing each modeling procedure comprises fitting the associated predictive model to the initial dataset;

(b) reducing the multi-model predictive value of the feature by shuffling values of the feature across respective observations included in the initial dataset, thereby generating a modified dataset;

(c) for each of the fitted predictive models;

(c1) determining a first accuracy score representing an accuracy with which the fitted model generates predictions for data in the initial dataset;

(c2) determining a second accuracy score representing an accuracy with which the fitted model generates predictions for data in the modified dataset in which the multi-model predictive value of the feature has been reduced; and

(c3) determining a model-specific predictive value of the feature based on the first and second accuracy scores of the fitted model; and

(d) determining, based on the model-specific predictive values of the feature, that the multi-model predictive value of the feature is low;

performing feature engineering on the initial dataset based on the multi-model predictive value of the feature, including pruning the feature having the low multi-model predictive value from the initial dataset, thereby generating a pruned dataset; and

building a predictive model for the prediction problem, including;

performing a plurality of predictive modeling procedures on the pruned dataset, selecting a fitted predictive model generated by the plurality of predictive modeling procedures, and deploying the selected predictive model to predict outcomes of the prediction problem without using the pruned feature.

2. The method of claim 1, further comprising:

prior to performing the one or more predictive modeling procedures, selecting the one or more predictive modeling procedures based on characteristics of the initial dataset, characteristics of the prediction problem, and/or characteristics of the feature.

3. The method of claim 1, wherein the one or more predictive modeling procedures comprise two or more modeling procedures selected from the group consisting of a random forest modeling procedure, a generalized additive modeling procedure, and a support vector machine modeling procedure.

4. The method of claim 1, wherein the one or more predictive modeling procedures comprise a first modeling procedure selected from a first family of modeling procedures and a second modeling procedure selected from a second family of modeling procedures.

5. The method of claim 1, further comprising:

prior to determining the second accuracy scores of the one or more predictive models, refitting the one or more predictive models to the modified dataset.

6. The method of claim 1, wherein the determined model-specific predictive value of the feature for a particular fitted model increases as the difference between the first accuracy score and the second accuracy score of the particular fitted model increases.

7. The method of claim 1, wherein the determined model-specific predictive value of the feature for a particular fitted model comprises a percentage difference between the first accuracy score and the second accuracy score of the particular fitted model, relative to the first accuracy score of the particular fitted model.

8. The method of claim 1, further comprising determining a model-independent predictive value of the feature.

9. The method of claim 8, wherein determining the model-independent predictive value of the feature comprises calculating a statistical measure of a center and/or a spread of a plurality of model-specific predictive values of the feature.

10. The method of claim 9, wherein determining the model-independent predictive value of the feature comprises calculating the statistical measure of the center of the plurality of model-specific predictive values, and wherein the statistical measure of the center is selected from the group consisting of a mean, a median, and a mode of the plurality of model-specific predictive values.

11. The method of claim 9, wherein determining the model-independent predictive value of the feature comprises calculating the statistical measure of the spread of the plurality of model-specific predictive values, and wherein the statistical measure of the spread is selected from the group consisting of a range, a variance, and a standard deviation of the plurality of model-specific predictive values.

12. The method of claim 8, wherein determining the model-independent predictive value of the feature comprises calculating a combination of the plurality of model-specific predictive values of the feature.

13. The method of claim 12, wherein calculating a combination of the plurality of model-specific predictive values comprises calculating a weighted combination of the plurality of model-specific predictive values.

14. The method of claim 13, wherein calculating the weighted combination of the plurality of model-specific predictive values comprises assigning respective weights to the plurality of model-specific predictive values, wherein the weight assigned to a particular model-specific predictive value corresponding to a particular fitted predictive model increases as the first accuracy score of the fitted predictive model increases.

15. The method of claim 1, wherein the feature is a first feature, and wherein the method further comprises:

determining a multi-model predictive value of a second feature of the initial dataset representing the prediction problem, including;

(bb) reducing the multi-model predictive value of the second feature by shuffling values of the second feature across respective observations included in the initial dataset, thereby generating a second modified dataset;

(cc) for each of the fitted predictive models;

(cc2) determining a third accuracy score representing an accuracy with which the fitted model generates predictions for data in the second modified dataset in which the multi-model predictive value of the second feature has been reduced; and

(cc3) determining a model-specific predictive value of the second feature based on the first and second accuracy scores of the fitted model; and

(dd) determining, based on the model-specific predictive values of the second feature, a multi-model predictive value of the feature.

16. The method of claim 1, wherein the feature is a first feature, wherein the pruned dataset includes a plurality of second features, and wherein the method further comprises determining model-specific predictive values of the second features of the pruned dataset by performing steps (b), (c2), and (c3) for each of the second features.

17. The method of claim 16, further comprising displaying, via a graphical user interface, graphical content identifying the second features of the pruned dataset and the model-specific predictive values of the second features.

18. The method of claim 16, wherein the one or more predictive modeling procedures are first modeling procedures including a particular modeling procedure associated with a particular predictive model, wherein the model-specific predictive values of the second features include particular model-specific predictive values of the second features that are specific to the particular predictive model, and wherein the plurality of predictive modeling procedures includes the particular modeling procedure associated with the particular predictive model.

19. The method of claim 18, wherein performing the particular predictive modeling procedure comprises performing feature engineering on the pruned dataset based on the particular model-specific predictive values of the second features.

20. The method of claim 19, wherein performing feature engineering comprises removing a particular second feature from the pruned dataset based on the particular second feature having a low model-specific predictive value.

21. The method of claim 20, further comprising determining that the model-specific predictive value of the particular second feature is low based on the model-specific predictive value of the particular second feature being lower than a threshold value and/or based on the model-specific predictive value of the particular second feature being in a specified percentile of the particular model-specific predictive values for the second features of the pruned dataset.

22. The method of claim 19, wherein the pruned dataset is a first pruned dataset, and wherein performing feature engineering comprises:

generating a derived feature based on two or more particular second features of the first pruned dataset having high model-specific predictive values; and

adding the derived feature to the first pruned dataset, thereby generating a second pruned dataset.

23. The method of claim 22, further comprising determining that the model-specific predictive values of the particular second features are high based on the model-specific predictive values of the particular second features being higher than a threshold value and/or based on the model-specific predictive values of the particular second features being in a specified percentile of the particular model-specific predictive values for the second features of the first pruned dataset.

24. The method of claim 22, wherein performing the particular predictive modeling procedure further comprises fitting the particular predictive model to the second pruned dataset, and wherein the method further comprises:

determining a first accuracy score of the fitted particular predictive model representing an accuracy with which the fitted particular model generates predictions for data in the second pruned dataset;

shuffling values of a particular feature of the second pruned dataset across respective observations included in the second pruned dataset, thereby generating a second modified dataset;

determining a second accuracy score of the fitted particular predictive model representing an accuracy with which the fitted particular model generates predictions for data in the second modified dataset; and

determining a model-specific predictive value of the particular feature for the fitted particular model based on the first and second accuracy scores of the fitted particular model.

25. The method of claim 18, further comprising, prior to performing the plurality of modeling procedures:

selecting the plurality of modeling procedures based on suitabilities of the selected modeling procedures for the prediction problem,

wherein a suitability of the particular predictive modeling procedure for the prediction problem is determined based, at least in part, on characteristics of one or more particular features of the pruned dataset having high model-specific predictive values for the particular predictive modeling procedure.

26. The method of claim 25, further comprising:

transmitting instructions to a plurality of processing nodes, the instructions comprising a resource allocation schedule allocating resources of the processing nodes for execution of the plurality of modeling procedures, the resource allocation schedule being based, at least in part, on the suitabilities of the plurality of modeling procedures for the prediction problem; and

receiving results of the execution of the plurality of modeling procedures by the plurality of processing nodes in accordance with the resource allocation schedule, wherein the results include predictive models generated by the plurality of modeling procedures, and/or scores of the generated models for data associated with the prediction problem, wherein the selected predictive model is selected from the generated models based, at least in part, on the score of the selected predictive model.

27. The method of claim 25, further comprising:

generating a blended predictive model by combining two or more of the generated predictive models; and

evaluating the blended predictive model.

28. The method of claim 1, wherein determining that the multi-model predictive value of the feature is low comprises:

determining the multi-model predictive value of the feature based on the model-specific predictive values of the feature; and

determining that the multi-model predictive value of the feature is less than a threshold predictive value.

29. A predictive modeling apparatus comprising:

a memory configured to store processor-executable instructions; and

a processor configured to execute the processor-executable instructions, wherein executing the processor-executable instructions causes the apparatus to perform steps including;

determining a multi-model predictive value of a feature of an initial dataset representing a prediction problem, wherein the initial dataset includes a plurality of observations and each observation includes respective values for a plurality of features, including;

(a) performing one or more predictive modeling procedures, wherein each of the predictive modeling procedures is associated with a different type of predictive model, wherein performing each modeling procedure comprises fitting the associated predictive model to the initial dataset;

(b) reducing the multi-model predictive value of the feature by shuffling values of the feature across respective observations included in the initial dataset, thereby generating a modified dataset;

(c) for each of the fitted predictive models;

(c1) determining a first accuracy score representing an accuracy with which the fitted model generates predictions for data in the initial dataset;

(c2) determining a second accuracy score representing an accuracy with which the fitted model generates predictions for data in the modified dataset in which the multi-model predictive value of the feature has been reduced; and

(c3) determining a model-specific predictive value of the feature based on the first and second accuracy scores of the fitted model; and

(d) determining, based on the model-specific predictive values of the feature, that the multi-model predictive value of the feature is low;

performing feature engineering on the initial dataset based on the multi-model predictive value of the feature, including pruning the feature having the low multi-model predictive value from the initial dataset, thereby generating a pruned dataset; and

building a predictive model for the prediction problem, including;

performing a plurality of predictive modeling procedures on the pruned dataset, selecting a fitted predictive model generated by the plurality of predictive modeling procedures, and deploying the selected predictive model to predict outcomes of the prediction problem without using the pruned feature.

30. An article of manufacture having computer-readable instructions stored thereon that, when executed by a processor, cause the processor to perform operations for building a predictive model, including:

determining a multi-model predictive value of a feature of an initial dataset representing a prediction problem, wherein the initial dataset includes a plurality of observations and each observation includes respective values for a plurality of features, including;

(a) performing one or more predictive modeling procedures, wherein each of the predictive modeling procedures is associated with a different type of predictive model, wherein performing each modeling procedure comprises fitting the associated predictive model to the initial dataset;

(b) reducing the multi-model predictive value of the feature by shuffling values of the feature across respective observations included in the initial dataset, thereby generating a modified dataset;

(c) for each of the fitted predictive models;

(c1) determining a first accuracy score representing an accuracy with which the fitted model generates predictions for data in the initial dataset;

(c2) determining a second accuracy score representing an accuracy with which the fitted model generates predictions for data in the modified dataset in which the multi-model predictive value of the feature has been reduced; and

(c3) determining a model-specific predictive value of the feature based on the first and second accuracy scores of the fitted model; and

(d) determining, based on the model-specific predictive values of the feature, that the multi-model predictive value of the feature is low;

performing feature engineering on the initial dataset based on the multi-model predictive value of the feature, including pruning the feature having the low multi-model predictive value from the initial dataset, thereby generating a pruned dataset; and

building a predictive model for the prediction problem, including;

performing a plurality of predictive modeling procedures on the pruned dataset, selecting a fitted predictive model generated by the plurality of predictive modeling procedures, and deploying the selected predictive model to predict outcomes of the prediction problem without using the pruned feature.