Systems and techniques for predictive data analytics

• US 9,659,254 B2
• Filed: 07/22/2016
• Issued: 05/23/2017
• Est. Priority Date: 05/23/2014
• Status: Active Grant

• Current Assignee: DataRobot, Inc.
• Original Assignee: DataRobot, Inc.
• Inventors: Achin, Jeremy, DeGodoy, Thomas, Owen, Timothy, Conort, Xavier
• Primary Examiner(s): SITIRICHE, LUIS A
• Application Number:
  US15/217,640
• Publication Number:
  US 20160364647A1
• Time in Patent Office: 305 Days
• Field of Search: None
• CPC Class Codes: G06F 9/5011 the resources being hardware resources other than CPUs, Servers and Terminals
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06Q 10/04 Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" market predictions or forecasting for commercial activities G06Q30/0202
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/067 Enterprise or organisation modelling

Abstract

Systems and techniques for predictive data analytics are described. In a method for selecting a predictive model for a prediction problem, the suitabilities of predictive modeling procedures for the prediction problem may be determined based on characteristics of the prediction problem and/or on attributes of the respective modeling procedures. A subset of the predictive modeling procedures may be selected based on the determined suitabilities of the selected modeling procedures for the prediction problem. A resource allocation schedule allocating computational resources for execution of the selected modeling procedures may be generated, based on the determined suitabilities of the selected modeling procedures for the prediction problem. Results of the execution of the selected modeling procedures in accordance with the resource allocation schedule may be obtained. A predictive model for the prediction problem may be selected based on those results.

Claims

1. A method for selecting a predictive model for a prediction problem, the method comprising:

determining suitabilities of a plurality of predictive modeling procedures for the prediction problem based, at least in part, on characteristics of the prediction problem and/or on attributes of the respective modeling procedures, wherein, for each of the predictive modeling procedures, the determined suitability of the predictive modeling procedure for the prediction problem comprises a value representing an estimated performance of a predictive model generated using the predictive modeling procedure and applied to the prediction problem;

selecting at least a subset of the predictive modeling procedures based, at least in part, on the determined suitabilities of the selected modeling procedures for the prediction problem;

transmitting instructions to a plurality of processing nodes, the instructions comprising a resource allocation schedule allocating resources of the processing nodes for execution of the selected modeling procedures based, at least in part, on the determined suitabilities of the selected modeling procedures for the prediction problem, wherein the resource allocation schedule allocates different portions of the resources of the processing nodes to different subsets of the selected modeling procedures in accordance with the determined suitabilities of the selected modeling procedures;

receiving results of the execution of the selected modeling procedures by the plurality of processing nodes in accordance with the resource allocation schedule, wherein the results include predictive models generated by the selected modeling procedures, and/or scores of the models for data associated with the prediction problem; and

selecting, from the predictive models, a predictive model for the prediction problem based, at least in part, on the score of the selected predictive model.

2. The method of claim 1, wherein determining the suitabilities of the plurality of predictive modeling procedures for the prediction problem comprises:

eliminating at least one predictive modeling procedure from consideration based on one or more relationships between the characteristics of the prediction problem and the attributes of the eliminated procedure, and/or

for at least one predictive modeling procedure, determining the value representing the estimated performance of a predictive model generated using the predictive modeling procedure based on one or more relationships between the characteristics of the prediction problem and the attributes of the predictive modeling procedure.

3. The method of claim 1, wherein selecting at least a subset of the predictive modeling procedures comprises:

selecting one or more predictive modeling procedures for which the value exceeds a threshold suitability value,

selecting one or more predictive modeling procedures for which the value is within a specified range of a highest value assigned to any of the predictive modeling procedures for the prediction problem, or

selecting a specified fraction of the predictive modeling procedures for which the value is highest.

4. The method of claim 3, further comprising determining the threshold suitability value based, at least in part, on an amount of processing resources available for execution of the selected modeling procedures.

5. The method of claim 1, wherein the allocated resources of the processing nodes comprise execution cycles of the processing nodes, execution time on the processing nodes, and/or computer-readable storage of the processing nodes.

6. The method of claim 1, further comprising receiving budget data indicating a temporal and/or resource budget for executing the selected modeling procedures, wherein the resource allocation schedule allocates resources of the processing nodes in accordance with the temporal and/or resource budget.

7. The method of claim 1, wherein the results of execution of the selected predictive modeling procedures include results of:

fitting the predictive models to a first portion of the data associated with the prediction problem, and testing the fitted models on a second portion of the data associated with the prediction problem.

8. The method of claim 7, wherein fitting the predictive models to the first portion of the data comprises tuning one or more parameters of the selected modeling procedures and/or one or more parameters of the predictive models.

9. The method of claim 1, wherein selecting a predictive model for the prediction problem based, at least in part, on the score of the selected model comprises:

selecting a predictive model having a score that exceeds a threshold score, or selecting a predictive model having a score within a specified range of a highest score of any of the predictive models.

10. The method of claim 1, further comprising:

generating a blended predictive model by combining two or more of the predictive models; and

evaluating the blended predictive model.

11. The method of claim 1, further comprising:

iteratively receiving the scores of the predictive models and re-determining the suitabilities of the selected predictive modeling procedures for the prediction problem based, at least in part, on the scores until a temporal and/or resource budget has been used or a score of at least one of the predictive models exceeds a threshold score.

12. A predictive modeling apparatus comprising:

a memory configured to store processor-executable instructions; and

a processor configured to execute the processor-executable instructions, wherein executing the processor-executable instructions causes the apparatus to perform a method comprising;

determining suitabilities of a plurality of predictive modeling procedures for a prediction problem based, at least in part, on characteristics of the prediction problem and/or on attributes of the respective modeling procedures, wherein, for each of the predictive modeling procedures, the determined suitability of the predictive modeling procedure for the prediction problem comprises a value representing an estimated performance of a predictive model generated using the predictive modeling procedure and applied to the prediction problem,

selecting at least a subset of the predictive modeling procedures based, at least in part, on the determined suitabilities of the selected modeling procedures for the prediction problem,

transmitting instructions to a plurality of processing nodes, the instructions comprising a resource allocation schedule allocating resources of the processing nodes for execution of the selected modeling procedures based, at least in part, on the determined suitabilities of the selected modeling procedures for the prediction problem, wherein the resource allocation schedule allocates different portions of the resources of the processing nodes to different subsets of the selected modeling procedures in accordance with the determined suitabilities of the selected modeling procedures,

receiving results of the execution of the selected modeling procedures by the plurality of processing nodes in accordance with the resource allocation schedule, wherein the results include predictive models generated by the selected modeling procedures, and/or scores of the models for data associated with the prediction problem, and

selecting, from the predictive models, a predictive model for the prediction problem based, at least in part, on the score of the selected predictive model.

13. The apparatus of claim 12, wherein determining the suitabilities of the plurality of predictive modeling procedures for the prediction problem comprises:

eliminating at least one predictive modeling procedure from consideration based on one or more relationships between the characteristics of the prediction problem and the attributes of the eliminated procedure, and/or

for at least one predictive modeling procedure, determining the value representing the estimated performance of a predictive model generated using the predictive modeling procedure based on one or more relationships between the characteristics of the prediction problem and the attributes of the predictive modeling procedure.

14. The apparatus of claim 12, wherein selecting at least a subset of the predictive modeling procedures comprises:

selecting one or more predictive modeling procedures for which the value exceeds a threshold suitability value,

selecting one or more predictive modeling procedures for which the value is within a specified range of a highest value assigned to any of the predictive modeling procedures for the prediction problem, or

selecting a specified fraction of the predictive modeling procedures for which the value is highest.

15. The apparatus of claim 14, wherein the method further comprises determining the threshold suitability value based, at least in part, on an amount of processing resources available for execution of the selected modeling procedures.

16. The apparatus of claim 12, wherein the allocated resources of the processing nodes comprise execution cycles of the processing nodes, execution time on the processing nodes, and/or computer-readable storage of the processing nodes.

17. The apparatus of claim 12, wherein the method further comprises receiving budget data indicating a temporal and/or resource budget for executing the selected modeling procedures, and wherein the resource allocation schedule allocates resources of the processing nodes in accordance with the temporal and/or resource budget.

18. The apparatus of claim 12, wherein the results of execution of the selected predictive modeling procedures include results of:

fitting the predictive models to a first portion of the data associated with the prediction problem, and testing the fitted models on a second portion of the data associated with the prediction problem.

19. The apparatus of claim 18, wherein fitting the predictive models to the first portion of the data comprises tuning one or more parameters of the selected modeling procedures and/or one or more parameters of the predictive models.

20. The apparatus of claim 12, wherein selecting a predictive model for the prediction problem based, at least in part, on the score of the selected model comprises:

selecting a predictive model having a score that exceeds a threshold score, or selecting a predictive model having a score within a specified range of a highest score of any of the predictive models.

21. The apparatus of claim 12, wherein the method further comprises:

generating a blended predictive model by combining two or more of the predictive models; and

evaluating the blended predictive model.

22. The apparatus of claim 12, wherein the method further comprises:

iteratively receiving the scores of the predictive models and re-determining the suitabilities of the selected predictive modeling procedures for the prediction problem based, at least in part, on the scores until a temporal and/or resource budget has been used or a score of at least one of the predictive models exceeds a threshold score.

23. A method for selecting a predictive model for a prediction problem, the method comprising:

determining suitabilities of a plurality of predictive modeling procedures for the prediction problem based, at least in part, on characteristics of the prediction problem and/or on attributes of the respective modeling procedures;

selecting at least a subset of the predictive modeling procedures based, at least in part, on the determined suitabilities of the selected modeling procedures for the prediction problem, wherein the selected modeling procedures include a first modeling procedure;

transmitting instructions to a plurality of processing nodes, the instructions comprising a resource allocation schedule allocating resources of the processing nodes for execution of the selected modeling procedures, the resource allocation schedule being based, at least in part, on the determined suitabilities of the selected modeling procedures for the prediction problem;

executing, by at least one of the processing nodes, the first modeling procedure, wherein executing the first modeling procedure includes;

generating training data and testing data from input data;

fitting a first predictive model to the training data; and

testing the fitted model on the testing data;

receiving results of the execution of the selected modeling procedures by the plurality of processing nodes in accordance with the resource allocation schedule, wherein the results include predictive models generated by the selected modeling procedures, and/or scores of the models for data associated with the prediction problem, wherein the predictive models include the first predictive model; and

selecting, from the predictive models, a predictive model for the prediction problem based, at least in part, on the score of the selected predictive model.

24. The method of claim 23, wherein the input data comprise at least one data set, wherein generating the training data comprises obtaining a first subset of the data set, and wherein generating the testing data comprises obtaining a second subset of the data set.

25. The method of claim 24, wherein executing the first predictive modeling procedure further includes performing cross-validation of the first predictive model.

26. The method of claim 25, wherein the training data are first training data, wherein the testing data are first testing data, wherein the fitted model is a first fitted model, and wherein performing the cross-validation of the first predictive model comprises:

(a) generating, from the data set, second training data and second testing data, wherein generating the second training data comprises obtaining a third subset of the data set, and wherein generating the second testing data comprises obtaining a fourth subset of the data set;

(b) fitting the first predictive model to the second training data to obtain a second fitted model; and

(c) testing the second fitted model on the second testing data.

27. The method of claim 26, wherein executing the first predictive modeling procedure further comprises partitioning the data set into a plurality of partitions including at least a first partition and a second partition.

28. The method of claim 27, wherein:

the first training data comprise the first partition of the data set;

the first testing data comprise all of the partitions of the data set except the first partition;

the second training data comprise the second partition of the data set; and

the second testing data comprise all of the partitions of the data set except the second partition.

29. The method of claim 27, wherein:

the first training data comprise a subset of the first partition of the data set;

the first testing data comprise respective subsets of all of the partitions of the data set except the first partition;

the second training data comprise a subset of the second partition of the data set; and

the second testing data comprise respective subsets of all of the partitions of the data set except the second partition.

30. The method of claim 26, wherein:

the input data comprise a first partition and a second partition,

the data set comprises the first partition of the input data, and

executing the first predictive modeling procedure further comprises testing the first and second fitted models on holdout data comprising the second partition of the input data,

wherein no predictive model is fitted to the holdout data.

31. The method of claim 23, wherein executing the first predictive modeling procedure further includes performing nested cross-validation of the first predictive model.

32. The method of claim 31, wherein:

the input data comprise at least one data set;

performing the nested cross-validation of the first predictive model comprises;

partitioning the data set into a first plurality of partitions of the data set including at least a first partition of the data set and a second partition of the data set, and

partitioning the first partition of the data set into a plurality of partitions of the first partition of the data set including at least a first partition of the first partition of the data set and a second partition of the first partition of the data set;

the training data comprise the first partition of the first partition of the data set; and

the testing data comprise all of the partitions of the first partition of the data set except the first partition of the first partition of the data set.

33. The method of claim 32, wherein the training data are first training data, the testing data are first testing data, the fitted model is a first fitted model, and performing the nested cross-validation of the first predictive model further comprises:

(a) generating, from the first partition of the data set, second training data and second testing data, wherein the second training data comprise the second partition of the first partition of the data set, and wherein the second testing data comprise a plurality of the partitions of the first partition of the data set other than the second partition of the first partition of the data set;

(b) fitting the first predictive model to the second training data to obtain a second fitted model; and

(c) testing the second fitted model on the second testing data.

34. The method of claim 33, wherein performing the nested cross-validation further includes:

testing the first fitted model and the second fitted model on the second partition of the data set; and

comparing the first fitted model to the second fitted model based on results of testing the first and second fitted models on the second partition of the data set.

35. The method of claim 23, further comprising deploying the fitted model.

36. The method of claim 35, wherein deploying the fitted model comprises:

generating a plurality of predictions by applying the fitted model to other data representing instances of the prediction problem, wherein the input data do not include the other data, and/or

caching a state of the fitted model in a memory.

37. The method of claim 35, wherein the fitted model has a first representation, and wherein deploying the fitted model comprises:

generating a second representation of the fitted model, wherein the second representation comprises a set of one or more conditional rules,

wherein the second representation of the fitted model is a machine executable representation, and wherein the set of one or more conditional rules comprises a set of one or more machine executable if-then statements.

38. The method of claim 35, wherein the input data are first input data, and wherein deploying the fitted model further comprises refreshing the fitted model based, at least in part, on second input data.

39. The method of claim 38, wherein the fitted model is a first fitted model, and wherein refreshing the fitted model based, at least in part, on the second input data comprises:

executing the first predictive modeling procedure on the second input data to generate a second fitted model; and

blending the first fitted model and the second fitted model to generate a refreshed predictive model.

40. The method of claim 38, wherein refreshing the fitted model based, at least in part, on the second input data comprises:

executing the first predictive modeling procedure on third input data comprising at least a portion of the first input data and at least a portion of the second input data to generate a refreshed predictive model.

41. A predictive modeling apparatus comprising:

a memory configured to store processor-executable instructions; and

a processor configured to execute the processor-executable instructions, wherein executing the processor-executable instructions causes the apparatus to perform operations comprising;

determining suitabilities of a plurality of predictive modeling procedures for a prediction problem based, at least in part, on characteristics of the prediction problem and/or on attributes of the respective modeling procedures;

selecting at least a subset of the predictive modeling procedures based, at least in part, on the determined suitabilities of the selected modeling procedures for the prediction problem, wherein the selected modeling procedures include a first modeling procedure;

transmitting instructions to a plurality of processing nodes, the instructions comprising a resource allocation schedule allocating resources of the processing nodes for execution of the selected modeling procedures, the resource allocation schedule being based, at least in part, on the determined suitabilities of the selected modeling procedures for the prediction problem;

executing, by at least one of the processing nodes, the first modeling procedure, wherein executing the first modeling procedure includes;

generating training data and testing data from input data;

fitting a first predictive model to the training data; and

testing the fitted model on the testing data;

receiving results of the execution of the selected modeling procedures by the plurality of processing nodes in accordance with the resource allocation schedule, wherein the results include predictive models generated by the selected modeling procedures, and/or scores of the models for data associated with the prediction problem, wherein the predictive models include the first predictive model; and

selecting, from the predictive models, a predictive model for the prediction problem based, at least in part, on the score of the selected predictive model.

42. The apparatus of claim 41, wherein the input data comprise at least one data set, wherein generating the training data comprises obtaining a first subset of the data set, and wherein generating the testing data comprises obtaining a second subset of the data set.

43. The apparatus of claim 42, wherein executing the first predictive modeling procedure further includes performing cross-validation of the first predictive model.

44. The apparatus of claim 43, wherein the training data are first training data, wherein the testing data are first testing data, wherein the fitted model is a first fitted model, and wherein performing the cross-validation of the first predictive model comprises:

(a) generating, from the data set, second training data and second testing data, wherein generating the second training data comprises obtaining a third subset of the data set, and wherein generating the second testing data comprises obtaining a fourth subset of the data set;

(b) fitting the first predictive model to the second training data to obtain a second fitted model; and

(c) testing the second fitted model on the second testing data.

45. The apparatus of claim 44, wherein executing the first predictive modeling procedure further comprises partitioning the data set into a plurality of partitions including at least a first partition and a second partition.

46. The apparatus of claim 45, wherein:

the first training data comprise the first partition of the data set;

the first testing data comprise all of the partitions of the data set except the first partition;

the second training data comprise the second partition of the data set; and

the second testing data comprise all of the partitions of the data set except the second partition.

47. The apparatus of claim 45, wherein:

the first training data comprise a subset of the first partition of the data set;

the first testing data comprise respective subsets of all of the partitions of the data set except the first partition;

the second training data comprise a subset of the second partition of the data set; and

the second testing data comprise respective subsets of all of the partitions of the data set except the second partition.

48. The apparatus of claim 44, wherein:

the input data comprise a first partition and a second partition,

the data set comprises the first partition of the input data,

executing the first predictive modeling procedure further comprises testing the first and second fitted models on holdout data comprising the second partition of the input data, and

no predictive model is fitted to the holdout data.

49. The apparatus of claim 41, wherein executing the first predictive modeling procedure further includes performing nested cross-validation of the first predictive model.

50. The apparatus of claim 49, wherein:

the input data comprise at least one data set;

performing the nested cross-validation of the first predictive model comprises;

partitioning the data set into a first plurality of partitions of the data set including at least a first partition of the data set and a second partition of the data set, and

partitioning the first partition of the data set into a plurality of partitions of the first partition of the data set including at least a first partition of the first partition of the data set and a second partition of the first partition of the data set;

the training data comprise the first partition of the first partition of the data set; and

the testing data comprise all of the partitions of the first partition of the data set except the first partition of the first partition of the data set.

51. The apparatus of claim 50, wherein the training data are first training data, the testing data are first testing data, the fitted model is a first fitted model, and performing the nested cross-validation of the first predictive model further comprises:

(a) generating, from the first partition of the data set, second training data and second testing data, wherein the second training data comprise the second partition of the first partition of the data set, and wherein the second testing data comprise a plurality of the partitions of the first partition of the data set other than the second partition of the first partition of the data set;

(b) fitting the first predictive model to the second training data to obtain a second fitted model; and

(c) testing the second fitted model on the second testing data.

52. The apparatus of claim 51, wherein performing the nested cross-validation further includes:

testing the first fitted model and the second fitted model on the second partition of the data set; and

comparing the first fitted model to the second fitted model based on results of testing the first and second fitted models on the second partition of the data set.

53. The apparatus of claim 41, wherein the operations further comprise deploying the fitted model.

54. The apparatus of claim 53, wherein deploying the fitted model comprises:

generating a plurality of predictions by applying the fitted model to other data representing instances of the prediction problem, wherein the input data do not include the other data, and/or

caching a state of the fitted model in a memory.

55. The apparatus of claim 53, wherein the fitted model has a first representation, and wherein deploying the fitted model comprises:

generating a second representation of the fitted model, wherein the second representation comprises a set of one or more conditional rules,

wherein the second representation of the fitted model is a machine executable representation, and wherein the set of one or more conditional rules comprises a set of one or more machine executable if-then statements.

56. The apparatus of claim 53, wherein the input data are first input data, and wherein deploying the fitted model further comprises refreshing the fitted model based, at least in part, on second input data.

57. The apparatus of claim 56, wherein the fitted model is a first fitted model, and wherein refreshing the fitted model based, at least in part, on the second input data comprises:

executing the first predictive modeling procedure on the second input data to generate a second fitted model; and

blending the first fitted model and the second fitted model to generate a refreshed predictive model.

58. The apparatus of claim 56, wherein refreshing the fitted model based, at least in part, on the second input data comprises:

executing the first predictive modeling procedure on third input data comprising at least a portion of the first input data and at least a portion of the second input data to generate a refreshed predictive model.