Systems for Second-Order Predictive Data Analytics, And Related Methods and Apparatus

• US 20200134489A1
• Filed: 12/20/2019
• Published: 04/30/2020
• Est. Priority Date: 05/23/2014
• Status: Active Grant

• Current Assignee: DataRobot, Inc.
• Original Assignee: DataRobot, Inc.
• Inventors: Achin, Jeremy, DeGodoy, Thomas, Owen, Timothy, Conort, Xavier, Yurgenson, Sergey, Steadman, Mark L., Koundry, Glen, Chua, Hon Nian
• Granted Patent: US 11,922,329 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 9/5011 the resources being hardware resources other than CPUs, Servers and Terminals
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06Q 10/04 Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" market predictions or forecasting for commercial activities G06Q30/0202
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling

Abstract

A predictive modeling method may include obtaining a fitted, first-order predictive model configured to predict values of output variables based on values of first input variables; and performing a second-order modeling procedure on the fitted, first-order model, which may include: generating input data including observations including observed values of second input variables and predicted values of the output variables; generating training data and testing data from the input data; generating a fitted second-order model of the fitted first-order model by fitting a second-order model to the training data; and testing the fitted, second-order model of the first-order model on the testing data. Each observation of the input data may be generated by (1) obtaining observed values of the second input variables, and (2) applying the first-order predictive model to corresponding observed values of the first input variables to generate the predicted values of the output variables.

Claims

1. A predictive modeling method comprising:

obtaining a fitted, first-order predictive model, wherein the first-order predictive model is configured to predict values of one or more output variables of a prediction problem based on values of one or more first input variables; and

performing a second-order predictive modeling procedure on the fitted, first-order model, wherein the second-order modeling procedure is associated with a second-order predictive model, and wherein performing the second-order predictive modeling procedure on the fitted, first-order model includes;

generating second-order input data including a plurality of second-order observations, wherein each second-order observation includes respective observed values of one or more second input variables and predicted values of the output variables, and wherein generating the second-order input data comprises, for each second-order observation;

obtaining the respective observed values of the second input variables and corresponding observed values of the first input variables, and applying the first-order predictive model to the corresponding observed values of the first input variables to generate the respective predicted values of the output variables,

generating, from the second-order input data, second-order training data and second-order testing data,

generating a fitted second-order predictive model of the fitted first-order model by fitting the second-order predictive model to the second-order training data, and

testing the fitted, second-order predictive model of the fitted first-order model on the second-order testing data.

2. The method of claim 1, wherein obtaining the fitted, first-order model comprises blending two fitted predictive models.

3. The method of claim 1, wherein the second-order predictive model is a RuleFit model, a generalized additive model, or a blend thereof.

12. The method of claim 1, wherein performing the second-order predictive modeling procedure further includes performing cross-validation or nested cross-validation of the second-order predictive model.

16. The method of claim 1, further comprising:

determining an accuracy score of each of the fitted predictive models, wherein the accuracy score of each fitted model represents an accuracy with which the fitted model predicts outcomes of one or more prediction problems.

18. The method of claim 16, wherein the accuracy score of the fitted second-order model exceeds the accuracy score of the fitted first-order model.

19. The method of claim 1, further comprising:

deploying the fitted second-order model, wherein deploying the fitted second-order model comprises generating a plurality of predictions by applying the fitted second-order model to other data representing instances of the prediction problem, wherein the second-order input data do not include the other data.

20. The method of claim 19, wherein the fitted second-order model comprises a set of one or more conditional rules, and wherein the set of one or more conditional rules comprises a set of one or more machine executable if-then statements.

21. The method of claim 19, wherein the second-order input data are first second-order input data, and wherein deploying the fitted second-order model further comprises refreshing the fitted second-order model based, at least in part, on second second-order input data.

22. The method of claim 21, wherein the fitted second-order model is a first fitted second-order model, and wherein refreshing the fitted second-order model based, at least in part, on the second second-order input data comprises:

generating, from the second second-order input data, second second-order training data and second second-order testing data;

generating a second fitted second-order model of the fitted first-order model by fitting the second-order predictive model to the second second-order training data;

testing the second fitted second-order model of the first-order model on the second second-order testing data; and

blending the first fitted second-order model and the second fitted second-order model to generate a refreshed second-order predictive model.

23. The method of claim 21, wherein the fitted second-order model is a first fitted second-order model, and wherein refreshing the fitted second-order model based, at least in part, on the second second-order input data comprises:

generating third second-order input data comprising at least a portion of the first second-order input data and at least a portion of the second second-order input data;

generating, from the third second-order input data, third second-order training data and third second-order testing data;

generating a second fitted second-order model of the fitted first-order model by fitting the second-order predictive model to the third second-order training data; and

testing the second fitted second-order model of the first-order model on the third second-order testing data.

27. The method of claim 1, wherein the second-order modeling procedure is one of a plurality of second-order modeling procedures, wherein the second-order predictive model is one of a plurality of second-predictive models, and wherein the method comprises performing the plurality of second-order modeling procedures on the fitted first-order model, thereby generating a plurality of fitted second-order models of the fitted first-order model.

28. The method of claim 27, further comprising:

determining an accuracy score of each of the fitted second-order predictive models, wherein the accuracy score of each fitted second-order model represents an accuracy with which the fitted second-order model predicts outcomes of one or more prediction problems;

determining which of the accuracy scores is highest; and

deploying the fitted second-order model with the highest accuracy score.

31. The method of claim 1, further comprising:

determining that the fitted second-order model is more computationally efficient than the fitted first-order model based on a measurement of a computational resource utilization of the fitted second-order model being less than a measurement of the computational resource utilization of the fitted first-order model.

32. The method of claim 31, wherein the measurements of the computational resource utilization of the fitted first-order model and the fitted second-order model comprise (1) measurements of respective execution times of the fitted first-order model and the fitted second-order model and/or (2) measurements of respective amounts of a physical resource used by the fitted first-order model and the fitted second-order model.

33. The method of claim 1, wherein the fitted first-order model is a time-series model.

4-11. -11. (canceled)

13-15. -15. (canceled)

17. (canceled)

24-26. -26. (canceled)

29. (canceled)

30. A predictive modeling apparatus comprising:

a memory configured to store a machine-executable module encoding a second-order predictive modeling procedure associated with a second-order predictive model, wherein the second-order predictive modeling procedure includes a plurality of tasks including at least one pre-processing task and at least one model-fitting task; and

at least one processor configured to execute the machine-executable module, wherein executing the machine-executable module causes the apparatus to perform the second-order predictive modeling procedure on a fitted, first-order predictive model, including;

performing the pre-processing task, including obtaining the fitted, first-order predictive model, wherein the first-order predictive model is configured to predict values of one or more output variables of a prediction problem based on values of one or more first input variables; and

performing the model-fitting task, including;

generating second-order input data including a plurality of second-order observations, wherein each second-order observation includes respective observed values of one or more second input variables and predicted values of the output variables, and wherein generating the second-order input data comprises, for each second-order observation;

obtaining the respective observed values of the second input variables and corresponding observed values of the first input variables, and applying the first-order predictive model to the corresponding observed values of the first input variables to generate the respective predicted values of the output variables,

generating, from the second-order input data, second-order training data and second-order testing data,

generating a fitted second-order predictive model of the fitted first-order model by

fitting the second-order predictive model to the second-order training data, and testing the fitted, second-order predictive model of the fitted first-order model on the second-order testing data.

34. The apparatus of claim 30, wherein the apparatus is further configured to determine that the fitted second-order model is more computationally efficient than the fitted first-order model based on a measurement of a computational resource utilization of the fitted second-order model being less than a measurement of the computational resource utilization of the fitted first-order model.

35. The apparatus of claim 34, wherein the measurements of the computational resource utilization of the fitted first-order model and the fitted second-order model comprise (1) measurements of respective execution times of the fitted first-order model and the fitted second-order model and/or (2) measurements of respective amounts of a physical resource used by the fitted first-order model and the fitted second-order model.

36. The apparatus of claim 30, wherein the fitted first-order model is a time-series model.

37. The apparatus of claim 30, wherein obtaining the fitted, first-order model comprises blending two fitted predictive models.

38. The apparatus of claim 30, wherein the second-order predictive model is a RuleFit model, a generalized additive model, or a blend thereof.

39. The apparatus of claim 30, wherein performing the second-order predictive modeling procedure further includes determining an accuracy score of each of the fitted predictive models, wherein the accuracy score of each fitted model represents an accuracy with which the fitted model predicts outcomes of one or more prediction problems.

40. The apparatus of claim 39, wherein the accuracy score of the fitted second-order model exceeds the accuracy score of the fitted first-order model.

41. The apparatus of claim 30, wherein the fitted second-order model comprises a set of one or more conditional rules, and wherein the set of one or more conditional rules comprises a set of one or more machine executable if-then statements.

42. The apparatus of claim 30, wherein:

the second-order modeling procedure is one of a plurality of second-order modeling procedures; and

the apparatus is configured to perform the second-order predictive modeling procedures on the fitted, first-order predictive model, thereby generating a plurality of fitted second-order models of the fitted first-order model.

43. The apparatus of claim 42, wherein the apparatus is further configured to:

determine an accuracy score of each of the fitted second-order predictive models, wherein the accuracy score of each fitted second-order model represents an accuracy with which the fitted second-order model predicts outcomes of one or more prediction problems;

determine which of the accuracy scores is highest; and

deploy the fitted second-order model with the highest accuracy score.

44. An article of manufacture having computer-readable instructions stored thereon that, when executed by a processor, cause the processor to perform operations including:

obtaining a fitted, first-order predictive model, wherein the first-order predictive model is configured to predict values of one or more output variables of a prediction problem based on values of one or more first input variables; and

performing a second-order predictive modeling procedure on the fitted, first-order model, wherein the second-order modeling procedure is associated with a second-order predictive model, and wherein performing the second-order predictive modeling procedure on the fitted, first-order model includes;

generating second-order input data including a plurality of second-order observations, wherein each second-order observation includes respective observed values of one or more second input variables and predicted values of the output variables, and wherein generating the second-order input data comprises, for each second-order observation;

obtaining the respective observed values of the second input variables and corresponding observed values of the first input variables, and applying the first-order predictive model to the corresponding observed values of the first input variables to generate the respective predicted values of the output variables,

generating, from the second-order input data, second-order training data and second-order testing data,

generating a fitted second-order predictive model of the fitted first-order model by fitting the second-order predictive model to the second-order training data, and

testing the fitted, second-order predictive model of the fitted first-order model on the second-order testing data.