Systems for time-series predictive data analytics, and related methods and apparatus

• US 10,496,927 B2
• Filed: 10/23/2017
• Issued: 12/03/2019
• Est. Priority Date: 05/23/2014
• Status: Active Grant

• Current Assignee: DataRobot, Inc.
• Original Assignee: DataRobot, Inc.
• Inventors: Achin, Jeremy, DeGodoy, Thomas, Owen, Timothy, Conort, Xavier, Yurgenson, Sergey, Steadman, Mark L., Koundry, Glen, Prettenhofer, Peter
• Primary Examiner(s): Sitiriche, Luis A
• Application Number:
  US15/790,803
• Publication Number:
  US 20180046926A1
• Time in Patent Office: 771 Days
• CPC Class Codes: G05B 17/02 electric
• CPC Class Codes: G06F 9/5011 the resources being hardware resources other than CPUs, Servers and Terminals
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 5/046 Forward inferencing; Production systems
• CPC Class Codes: G06Q 10/04 Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" market predictions or forecasting for commercial activities G06Q30/0202
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: Y02P 90/80 Management or planning

Abstract

A predictive modeling method may include determining a time interval of time-series data; identifying one or more variables of the data as targets; determining a forecast range and a skip range associated with a prediction problem represented by the data; generating training data and testing data from the time-series data; fitting a predictive model to the training data; and testing the fitted model on the testing data. The forecast range may indicate a duration of a period for which values of the targets are to be predicted. The skip range may indicate a temporal lag between the time period corresponding to the data used to make predictions and the time period corresponding to the predictions. The skip range may separate input data subsets representing model inputs from subsets representing model outputs, and separate test data subsets representing model inputs from subsets representing validation data.

Claims

1. A predictive modeling method comprising:

performing, with a processing device, a predictive modeling procedure, including;

(a) obtaining time-series data including one or more data sets, wherein each data set includes a plurality of observations, wherein each observation includes (1) an indication of a time associated with the observation and (2) respective values of one or more variables;

(b) identifying one or more of the variables as targets, and identifying zero or more other variables as features;

setting values of a plurality of adjustable parameters of the predictive modeling procedure, including;

(c) determining a value of a time interval parameter of the time-series data; and

(d) determining a value of a forecast range parameter and a value of a skip range parameter, wherein the forecast range parameter value indicates a duration of a period for which values of the targets are to be predicted, and wherein the skip range parameter value indicates a temporal lag between a time associated with an earliest prediction and a time associated with a latest observation upon which the earliest prediction is based;

segmenting the time-series data into training data and testing data based, at least in part, on the values of the forecast range parameter and the skip range parameter, including;

(e) generating the training data from the time-series data, wherein the training data include a first subset of the observations of at least one of the data sets, wherein the first subset of the observations includes training-input and training-output collections of the observations, wherein the times associated with the observations in the training-input and training-output collections correspond, respectively, to a training-input time range and a training-output time range, wherein the skip range parameter value separates an end of the training-input time range from a beginning of the training-output time range, and wherein a duration of the training-output time range is at least as long as the forecast range parameter value; and

(f) generating the testing data from the time-series data, wherein the testing data include a second subset of the observations of at least one of the data sets, wherein the second subset of the observations includes testing-input and testing-validation collections of the observations, wherein the times associated with the observations in the testing-input and testing-validation collections correspond, respectively, to a testing-input time range and a testing-validation time range, wherein the skip range parameter value separates an end of the testing-input time range from a beginning of the testing-validation time range, and wherein a duration of the testing-validation time range is at least as long as the forecast range parameter value; and

adapting a predictive model to solve a prediction problem represented by the time-series data and the values of the forecast range and skip range parameters, including;

(g) fitting the predictive model to the training data; and

(h) testing the fitted model on the testing data.

2. The method of claim 1, wherein the value of the time interval parameter of the time-series data is determined based, at least in part, on the times associated with at least a subset of the observations included in at least one of the data sets.

3. The method of claim 2, wherein determining the value of the time interval parameter of the time-series data comprises:

for each of the data sets, determining a respective time interval of the data set; and

determining that the time intervals of at least two of the data sets are different,

wherein the time interval parameter value of the time-series data is determined based, at least in part, on (1) respective proportions of the observations included in each of the data sets, and/or (2) the respective time intervals of each of the data sets.

4. The method of claim 3, wherein determining the respective time interval of the data set comprises:

determining respective time periods between each pair of successive observations included in the data set;

if the time periods between the pairs of successive observations exhibit a plurality of non-uniform durations, the time interval of the data set is determined based, at least in part, on (1) respective proportions of the pairs of successive observations exhibiting each of the non-uniform durations, and/or (2) the durations of the time periods; and

if the time periods between the pairs of successive observations are of uniform duration, the time interval of the data set is the duration of each of the time periods.

5. The method of claim 3, wherein the time interval parameter value of the time-series data is a shortest time interval that is an integer multiple of the respective time intervals of each of the data sets.

6. The method of claim 5, further comprising:

for each data set, if the time interval of the data set is shorter than the time interval parameter value of the time-series data, down-sampling the observations of the data set, thereby converting the time interval of the data set to the time interval parameter value of the time-series data.

7. The method of claim 6, wherein down-sampling the observations of the data set comprises, for each instance of the time interval parameter value of the time-series data in a time period corresponding to the data set:

identifying all observations in the data set associated with times corresponding to the respective instance of the time interval parameter value of the time-series data;

aggregating the identified observations to generate an aggregate observation; and

replacing the identified observations in the data set with the aggregate observation.

8. The method of claim 7, wherein aggregating the identified observations comprises setting a value of each variable in the aggregate observation to (1) the corresponding variable value included in an earliest of the identified observations, (2) the corresponding variable value included in a latest of the identified observations, (3) a greatest value of the corresponding variable values included in the identified observations, (4) a least value of the corresponding variable values included in the identified observations, (5) an average of the corresponding variable values included in the identified observations, or (6) a value of a function of the corresponding variable values included in the identified observations.

9. The method of claim 7, wherein a number of the identified observations corresponding to the instance of the time interval parameter value of the time-series data is equal to a ratio between the time interval parameter value of the time-series data and the time interval of the data set.

10. The method of claim 3, wherein the time interval parameter value of the time-series data is selected from a group consisting of the time intervals of the data sets.

11. The method of claim 10, wherein the data sets include a first data set exhibiting a first time interval and a second data set exhibiting a second time interval greater than the first time interval, wherein the value of the time interval parameter of the time-series data is set to represent the second time interval, and wherein the method further comprises:

down-sampling the observations of the first data set, thereby converting the time interval of the first data set to the time interval represented by the time interval parameter value of the time-series data.

12. The method of claim 3, wherein the value of the time interval parameter of the time-series data represents a time interval that differs from each of the time intervals of the data sets.

13. The method of claim 2, wherein determining the value of the time interval parameter of the time-series data comprises:

for each of the data sets, determining a respective time interval of the data set;

determining that the time intervals of the data sets are uniform; and

setting the value of the time interval parameter of the time-series data to represent the time interval of the data sets.

14. The method of claim 13, wherein determining the respective time interval of the data set comprises:

for one or more pairs of successive observations included in the data set, determining a respective time period between the successive observations;

determining that the time periods between the successive pairs of observations are uniform; and

setting the respective time interval of the data set to the time period between the successive pairs of observations.

15. The method of claim 1, wherein at least a group of the observations of the time-series data include respective values of a first variable, and wherein the method further comprises, prior to fitting the predictive model to the training data and testing the fitted model on the testing data, converting the first variable from a first time reference frame to a second time reference frame, including:

determining that the values of the first variable comprise time values;

for each observation in the group, generating a respective value of a second variable, wherein the value of the second variable comprises an offset between the time value of the first variable and a reference time value; and

adding the values of the second variable to the respective observations in the group.

16. The method of claim 15, wherein converting the first variable from the first time reference frame to the second time reference frame further includes:

removing the values of the first variable from the observations in the group.

17. The method of claim 15, wherein the reference time comprises a date of an event.

18. The method of claim 17, wherein the event comprises a birth, a wedding, a graduation from a school, a commencement of employment for an employer, or a commencement of work in a particular position.

19. The method of claim 1, wherein the forecast range parameter value is determined based, at least in part, on (1) the time interval parameter value of the time-series data, (2) a number of observations included in the time-series data, (3) a time period corresponding to the time-series data, and/or (4) a natural time period selected from the group consisting of microseconds, milliseconds, seconds, minutes, hours, days, weeks, months, quarters, seasons, years, decades, centuries, and millennia.

20. The method of claim 19, wherein the forecast range parameter value is an integer multiple of the time interval parameter value of the time-series data.

21. The method of claim 20, wherein a time period between times associated with successive predictions in a forecast range corresponding to the forecast range parameter value is equal to the time interval parameter value of the time-series data.

22. The method of claim 1, wherein the skip range parameter value is determined based, at least in part, on latency in collection of the time-series data, latency in communication of the time-series data, latency in analyzing the time-series data, latency in communication of analyses of the time series-data, and/or latency of implementing actions based on the analyses of the time series-data.

23. The method of claim 1, wherein segmenting the time-series data into training data and testing data further comprises determining a duration of the training-input time range based, at least in part, on a total number of observations included in the time-series data, an amount of variation in values of at least one of the variables over time, an amount of seasonal variation in values of at least one of the variables, a consistency of variation in values of at least one of the variables over a plurality of time periods, and/or the forecast range parameter value.

24. The method of claim 23, wherein fitting the predictive model to the training data comprises fitting the predictive model to a subset of the training data corresponding to a portion of the training-input time range, wherein the portion of the training-input time range starts at a time subsequent to a starting time of the training-input time range and ends at an ending time of the training-input time range.

25. The method of claim 24, wherein a duration of the portion of the training-input time range is an integer multiple of the forecast range parameter value.

26. The method of claim 1, wherein adapting the predictive model to solve the prediction problem further comprises performing cross-validation of the predictive model.

27. The method of claim 26, wherein the training data are first training data, wherein the testing data are first testing data, wherein the fitted model is a first fitted model, and wherein performing the cross-validation of the predictive model comprises:

(i) generating second training data and second testing data from the time-series data, wherein the second training data include a third subset of the observations of at least one of the data sets, and wherein the second testing data include a fourth subset of the observations of at least one of the data sets;

(j) fitting the predictive model to the second training data to obtain a second fitted model; and

(k) testing the second fitted model on the second testing data.

28. The method of claim 27, wherein the first subset of observations corresponds to a sliding training window covering a first range of training times and each observation included in the first subset is associated with a time within the first range of training times, wherein the third subset of observations corresponds to the sliding training window covering a second range of training times and each observation included in the third subset is associated with a time within the second range of training times, and wherein an earliest time in the first range of training times is earlier than an earliest time in the second range of training times.

29. The method of claim 28, wherein the second subset of observations corresponds to a sliding testing window covering a first range of testing times and each observation included in the second subset is associated with a time within the first range of testing times, wherein the fourth subset of observations corresponds to the sliding testing window covering a second range of testing times and each observation included in the fourth subset is associated with a time within the second range of testing times, and wherein an earliest time in the first range of testing times is earlier than an earliest time in the second range of testing times.

30. The method of claim 29, wherein the first testing time range partially overlaps the second training time range.

31. The method of claim 30, wherein the second testing time range does not overlap any portion of the first training time range, and does not overlap any portion of the second training time range.

32. The method of claim 27, further comprising partitioning the time-series data into a plurality of partitions including at least a first partition and a second partition.

33. The method of claim 32, wherein partitioning the time-series data into a plurality of partitions comprises assigning each of the data sets to a corresponding partition.

34. The method of claim 32, wherein partitioning the time-series data into a plurality of partitions comprises temporally partitioning the time-series data, wherein each of the partitions corresponds to a respective portion of a time period associated with the time-series data, and wherein each observation included in the time-series data is assigned to the partition corresponding to the portion of the time period that matches the time associated with the observation.

35. The method of claim 32, wherein:

the first training data comprise a subset of the observations included in the first partition of the time-series data;

the first testing data comprise respective subsets of the observations included in all of the partitions of the time-series data except the first partition;

the second training data comprise a subset of the observations included in the second partition of the time-series data; and

the second testing data comprise respective subsets of the observations included in all of the partitions of the time-series data except the second partition.

36. The method of claim 27, wherein a first partition of the time-series data includes the first and second training data and the first and second testing data, wherein a second partition of the time-series data includes holdout data, and wherein adapting the predictive model to solve the prediction problem further comprises:

testing the first and second fitted models on the holdout data.

37. The method of claim 36, wherein no predictive model is fitted to the holdout data.

38. The method of claim 1, further comprising:

performing nested cross-validation of the predictive model.

39. The method of claim 38, wherein performing the nested cross-validation of the predictive model comprises:

partitioning the time-series data into a first plurality of partitions including at least a first partition of the time-series data and a second partition of the time-series data; and

partitioning the first partition of the time-series data into a plurality of partitions of the first partition of the time-series data including at least a first partition of the first partition of the time-series data and a second partition of the first partition of the time-series data,

wherein the training data comprise the first partition of the first partition of the time-series data, and wherein the testing data comprise at least a plurality of the partitions of the first partition of the time-series data other than the first partition of the first partition of the time-series data.

40. The method of claim 39, wherein the training data are first training data, the testing data are first testing data, the fitted model is a first fitted model, and performing the nested cross-validation of the predictive model further comprises:

(i) generating, from the first partition of the time-series data, second training data and second testing data, wherein the second training data comprise the second partition of the first partition of the time-series data, and wherein the second testing data comprise at least a plurality of the partitions of the first partition of the data set other than the second partition of the first partition of the time-series data;

(j) fitting the predictive model to the second training data to obtain a second fitted model; and

(k) testing the second fitted model on the second testing data.

41. The method of claim 40, wherein performing the nested cross-validation further comprises:

testing the first fitted model and the second fitted model on the second partition of the time-series data; and

comparing the first fitted model to the second fitted model based on results of testing the first and second fitted models on the second partition of the time-series data.

42. The method of claim 1, further comprising deploying the fitted model, wherein the time-series data are first-time series data, and wherein deploying the fitted model includes:

obtaining second time-series data representing one or more instances of the prediction problem, wherein at least a portion of the second time-series data corresponds to a time period more recent than any portion of the first time-series data; and

generating at least one prediction associated with a future time by applying the fitted model to the second time-series data.

43. The method of claim 42, further comprising refreshing the fitted model, including:

performing the predictive modeling procedure on third time-series data comprising at least a portion of the first time-series data and at least a portion of the second time-series data, thereby generating a refreshed model.

44. The method of claim 1, wherein the variables include a first variable and a second variable, and wherein the method further comprises:

determining that changes in the values of the first and second variables are correlated, with a temporal lag between the changes in the value of the first variable and the correlated changes in the value of the second variable; and

displaying, via a graphical user interface, graphical content indicating a duration of the temporal lag between the changes in the value of the first variable and the correlated changes in the value of the second variable.

45. The method of claim 1, wherein adapting the predictive model to solve the prediction problem further comprises down-sampling the training data prior to fitting the predictive model to the training data.

46. The method of claim 45, wherein down-sampling the training data comprises:

removing, from the training data, all observations obtained from at least one of the data sets.

47. The method of claim 45, wherein down-sampling the training data comprises:

setting a value of a down-sampled time interval parameter of the training data to represent an integer multiple of the time interval parameter value of the time series data; and

for each instance of the down-sampled time interval parameter value of the training data;

identifying all observations in the training data associated with times corresponding to the respective instance of the down-sampled time interval parameter value of the training data,

aggregating the identified observations to generate an aggregate observation, and

replacing the identified observations in the training data with the aggregate observation.

48. The method of claim 1, wherein adapting the predictive model to solve the prediction problem further comprises down-sampling the testing data prior to testing the fitted model on the testing data.

49. The method of claim 1, further comprising:

determining suitabilities of a plurality of predictive modeling procedures for the prediction problem based, at least in part, on characteristics of the prediction problem and/or on attributes of the respective predictive modeling procedures;

selecting one or more predictive modeling procedures from the plurality of predictive modeling procedures based on the determined suitabilities of the selected modeling procedures for the prediction problem; and

performing the one or more predictive modeling procedures.

50. The method of claim 49, wherein performing the one or more predictive modeling procedures comprises:

transmitting instructions to a plurality of processing nodes, the instructions comprising a resource allocation schedule allocating resources of the processing nodes for execution of the selected modeling procedures, the resource allocation schedule being based, at least in part, on the suitabilities of the selected modeling procedures for the prediction problem;

receiving results of the execution of the selected modeling procedures by the plurality of processing nodes in accordance with the resource allocation schedule, wherein the results include predictive models generated by the selected modeling procedures, and/or scores of the generated models for time-series data associated with the prediction problem; and

selecting, from the generated models, a predictive model for the prediction problem based, at least in part, on the score of the selected predictive model.

51. The method of claim 1, further comprising:

generating a blended predictive model by blending the fitted model with another fitted model.

52. The method of claim 1, further comprising deploying the fitted model, wherein the time-series data are first time-series data, and wherein deploying the fitted model comprises refreshing the fitted model based, at least in part, on second time-series data.

53. The method of claim 52, wherein the fitted model is a first fitted model, and wherein refreshing the fitted model based, at least in part, on the second time-series data comprises:

performing the predictive modeling procedure on the second time-series data to generate a second fitted model; and

blending the first fitted model and the second fitted model to generate a refreshed predictive model.

54. The method of claim 1, further comprising deploying the fitted model, wherein the fitted model is deployed to one or more servers, wherein other fitted models are also deployed to the one or more servers, and wherein prediction requests to the fitted model and the other fitted models are allocated among the servers based, at least in part, on (1) an estimate of an amount of time used by each of the fitted models to generate a prediction, and/or (2) an estimate of a frequency with which prediction requests for each of the fitted models are received.

55. The method of claim 54, wherein each prediction request is assigned to a respective thread, wherein each prediction request has an associated latency-sensitivity value, and wherein a number of threads executing on a particular server is determined based, at least in part, on the latency-sensitivity values of the threads executing on the particular server.

56. The method of claim 1, further comprising:

determining a value of a metric that indicates an interaction strength of two or more of the features included in the time-series data; and

if the value of the metric exceeds a threshold value, generating time-series values of a new feature based on the values of the two or more features and adding the new feature to the time-series data.

57. The method of claim 1, further comprising:

determining a time resolution of the time-series data.

58. The method of claim 1, wherein the targets are identified based on user input.

59. The method of claim 1, further comprising:

determining, for the fitted model, model-specific predictive values of one or more of the features of the time-series data.

60. The method of claim 59, further comprising:

based at least in part on the model-specific predictive values of the features, performing at least one action selected from the group consisting of;

pruning a feature from the time-series data, creating a derived feature from two or more features in the time-series data and adding the derived feature to the time-series data, blending the predictive model with another predictive model, and/or allocating resources during a process of evaluating suitabilities of predictive modeling procedures for the prediction problem.

61. A predictive modeling apparatus comprising:

a memory configured to store a machine-executable module encoding a predictive modeling procedure, wherein the predictive modeling procedure includes a plurality of tasks including at least one pre-processing task and at least one model-fitting task; and

at least one processor configured to execute the machine-executable module, wherein executing the machine-executable module causes the apparatus to perform the predictive modeling procedure, including;

performing the pre-processing task, including;

(a) obtaining time-series data including one or more data sets, wherein each data set includes a plurality of observations, wherein each observation includes (1) an indication of a time associated with the observation and (2) respective values of one or more variables,

(b) identifying one or more of the variables as targets, and identifying zero or more other variables as features, and

setting values of a plurality of adjustable parameters of the predictive modeling procedure, including;

(c) determining a value of a time interval parameter of the time-series data, and

(d) determining a value of a forecast range and a value of a skip range parameter, wherein the forecast range parameter value indicates a duration of a period for which values of the targets are to be predicted, and wherein the skip range parameter value indicates a temporal lag between a time associated with an earliest prediction and a time associated with a latest observation upon which the earliest prediction is based; and

performing the model-fitting task, including;

segmenting the time-series data into training data and testing data based, at least in part, on the values of the forecast range parameter and the skip range parameter, including;

(e) generating the training data from the time-series data, wherein the training data include a first subset of the observations of at least one of the data sets, wherein the first subset of the observations includes training-input and training-output collections of the observations, wherein the times associated with the observations in the training-input and training-output collections correspond, respectively, to a training-input time range and a training-output time range, wherein the skip range parameter value separates an end of the training-input time range from a beginning of the training-output time range, and wherein a duration of the training-output time range is at least as long as the forecast range parameter value, and

(f) generating the testing data from the time-series data, wherein the testing data include a second subset of the observations of at least one of the data sets, wherein the second subset of the observations includes testing-input and testing-validation collections of the observations, wherein the times associated with the observations in the testing-input and testing-validation collections correspond, respectively, to a testing-input time range and a testing-validation time range, wherein the skip range parameter value separates an end of the testing-input time range from a beginning of the testing-validation time range, and wherein a duration of the testing-validation time range is at least as long as the forecast range parameter value, and

adapting a predictive model to solve a prediction problem represented by the time-series data and the values of the forecast range and skip range parameters, including;

(g) fitting the predictive model to the training data, and

(h) testing the fitted model on the testing data.

62. The apparatus of claim 61, wherein the machine-executable module includes a directed graph representing dependencies between the tasks.

63. The apparatus of claim 61, wherein the value of the time interval parameter of the time-series data is determined based, at least in part, on the times associated with at least a subset of the observations included in at least one of the data sets.

64. The apparatus of claim 63, wherein determining the value of the time interval parameter of the time-series data comprises:

for each of the data sets, determining a respective time interval of the data set; and

determining that the time intervals of at least two of the data sets are different,

wherein the time interval parameter value of the time-series data is determined based, at least in part, on (1) respective proportions of the observations included in each of the data sets, and/or (2) the respective time intervals of each of the data sets.

65. The apparatus of claim 64, wherein determining the respective time interval of the data set comprises:

determining respective time periods between each pair of successive observations included in the data set;

if the time periods between the pairs of successive observations exhibit a plurality of non-uniform durations, the time interval of the data set is determined based, at least in part, on (1) respective proportions of the pairs of successive observations exhibiting each of the non-uniform durations, and/or (2) the durations of the time periods; and

if the time periods between the pairs of successive observations are of uniform duration, the time interval of the data set is the duration of each of the time periods.

66. The apparatus of claim 64, wherein the time interval parameter value of the time-series data is a shortest time interval that is an integer multiple of the respective time intervals of each of the data sets.

67. The apparatus of claim 66, wherein performing the pre-processing task further includes:

for each data set, if the time interval of the data set is shorter than the time interval parameter value of the time-series data, down-sampling the observations of the data set, thereby converting the time interval of the data set to the time interval parameter value of the time-series data.

68. The apparatus of claim 67, wherein down-sampling the observations of the data set comprises, for each instance of the time interval parameter value of the time-series data in a time period corresponding to the data set:

identifying all observations in the data set associated with times corresponding to the respective instance of the time interval parameter value of the time-series data;

aggregating the identified observations to generate an aggregate observation; and

replacing the identified observations in the data set with the aggregate observation.

69. The apparatus of claim 68, wherein aggregating the identified observations comprises setting a value of each variable in the aggregate observation to (1) the corresponding variable value included in an earliest of the identified observations, (2) the corresponding variable value included in a latest of the identified observations, (3) a greatest value of the corresponding variable values included in the identified observations, (4) a least value of the corresponding variable values included in the identified observations, (5) an average of the corresponding variable values included in the identified observations, or (6) a value of a function of the corresponding variable values included in the identified observations.

70. The apparatus of claim 68, wherein a number of the identified observations corresponding to the instance of the time interval parameter value of the time-series data is equal to a ratio between the time interval parameter value of the time-series data and the time interval of the data set.

71. The apparatus of claim 64, wherein the time interval parameter value of the time-series data is selected from a group consisting of the time intervals of the data sets.

72. The apparatus of claim 71, wherein the data sets include a first data set exhibiting a first time interval and a second data set exhibiting a second time interval greater than the first time interval, wherein the value of the time interval parameter of the time-series data is set to represent the second time interval, and wherein performing the pre-processing task further comprises further comprises:

down-sampling the observations of the first data set, thereby converting the time interval of the first data set to the time interval represented by the time interval parameter value of the time-series data.

73. The apparatus of claim 63, wherein determining the value of the time interval parameter of the time-series data comprises:

for each of the data sets, determining a respective time interval of the data set;

determining that the time intervals of the data sets are uniform; and

setting the value of the time interval parameter of the time-series data to represent the time interval of the data sets.

74. The apparatus of claim 73, wherein determining the respective time interval of the data set comprises:

for one or more pairs of successive observations included in the data set, determining a respective time period between the successive observations;

determining that the time periods between the successive pairs of observations are uniform; and

setting the respective time interval of the data set to the time period between the successive pairs of observations.

75. The apparatus of claim 61, wherein at least a group of the observations of the time-series data include respective values of a first variable, and wherein performing the pre-processing task further comprises converting the first variable from a first time reference frame to a second time reference frame, including:

determining that the values of the first variable comprise time values;

for each observation in the group, generating a respective value of a second variable, wherein the value of the second variable comprises an offset between the time value of the first variable and a reference time value; and

adding the values of the second variable to the respective observations in the group.

76. The apparatus of claim 75, wherein:

converting the first variable from the first time reference frame to the second time reference frame further includes removing the values of the first variable from the observations in the group; and

/or

the reference time comprises a date of an event, wherein the event comprises a birth, a wedding, a graduation from a school, a commencement of employment for an employer, or a commencement of work in a particular position.

77. The apparatus of claim 61, wherein the forecast range parameter value is determined based, at least in part, on (1) the time interval parameter value of the time-series data, (2) a number of observations included in the time-series data, (3) a time period corresponding to the time-series data, and/or (4) a natural time period selected from the group consisting of microseconds, milliseconds, seconds, minutes, hours, days, weeks, months, quarters, seasons, years, decades, centuries, and millennia.

78. The apparatus of claim 77, wherein the forecast range parameter value is an integer multiple of the time interval parameter value of the time-series data.

79. The apparatus of claim 78, wherein a time period between times associated with successive predictions in a forecast range corresponding to the forecast range parameter value is equal to the time interval parameter value of the time-series data.

80. The apparatus of claim 61, wherein the skip range parameter value is determined based, at least in part, on latency in collection of the time-series data, latency in communication of the time-series data, latency in analyzing the time-series data, latency in communication of analyses of the time series-data, and/or latency of implementing actions based on the analyses of the time series-data.

81. The apparatus of claim 61, wherein segmenting the time-series data into training data and testing data further comprises determining a duration of the training-input time range based, at least in part, on a total number of observations included in the time-series data, an amount of variation in values of at least one of the variables over time, an amount of seasonal variation in values of at least one of the variables, a consistency of variation in values of at least one of the variables over a plurality of time periods, and/or the forecast range parameter value.

82. The apparatus of claim 81, wherein fitting the predictive model to the training data comprises fitting the predictive model to a subset of the training data corresponding to a portion of the training-input time range, wherein the portion of the training-input time range starts at a time subsequent to a starting time of the training-input time range and ends at an ending time of the training-input time range.

83. The apparatus of claim 82, wherein a duration of the portion of the training-input time range is an integer multiple of the forecast range parameter value.

84. The apparatus of claim 61, wherein adapting the predictive model to solve the prediction problem further comprises performing cross-validation of the predictive model.

85. The apparatus of claim 84, wherein the training data are first training data, wherein the testing data are first testing data, wherein the fitted model is a first fitted model, and wherein performing the cross-validation of the predictive model comprises:

(i) generating second training data and second testing data from the time-series data, wherein the second training data include a third subset of the observations of at least one of the data sets, and wherein the second testing data include a fourth subset of the observations of at least one of the data sets;

(j) fitting the predictive model to the second training data to obtain a second fitted model; and

(k) testing the second fitted model on the second testing data.

86. The apparatus of claim 85, wherein the first subset of observations corresponds to a sliding training window covering a first range of training times and each observation included in the first subset is associated with a time within the first range of training times, wherein the third subset of observations corresponds to the sliding training window covering a second range of training times and each observation included in the third subset is associated with a time within the second range of training times, and wherein an earliest time in the first range of training times is earlier than an earliest time in the second range of training times.

87. The apparatus of claim 86, wherein the second subset of observations corresponds to a sliding testing window covering a first range of testing times and each observation included in the second subset is associated with a time within the first range of testing times, wherein the fourth subset of observations corresponds to the sliding testing window covering a second range of testing times and each observation included in the fourth subset is associated with a time within the second range of testing times, and wherein an earliest time in the first range of testing times is earlier than an earliest time in the second range of testing times.

88. The apparatus of claim 87, wherein:

the first testing time range partially overlaps the second training time range; and

/or

the second testing time range does not overlap any portion of the first training time range, and does not overlap any portion of the second training time range.

89. The apparatus of claim 85, wherein performing the model-fitting task further comprises partitioning the time-series data into a plurality of partitions including at least a first partition and a second partition.

90. The apparatus of claim 89, wherein:

(1) partitioning the time-series data into a plurality of partitions comprises assigning each of the data sets to a corresponding partition;

(2) partitioning the time-series data into a plurality of partitions comprises temporally partitioning the time-series data, wherein each of the partitions corresponds to a respective portion of a time period associated with the time-series data, and wherein each observation included in the time-series data is assigned to the partition corresponding to the portion of the time period that matches the time associated with the observation;

or

(3) the first training data comprise a subset of the observations included in the first partition of the time-series data, the first testing data comprise respective subsets of the observations included in all of the partitions of the time-series data except the first partition, the second training data comprise a subset of the observations included in the second partition of the time-series data, and the second testing data comprise respective subsets of the observations included in all of the partitions of the time-series data except the second partition.

91. The apparatus of claim 85, wherein a first partition of the time-series data includes the first and second training data and the first and second testing data, wherein a second partition of the time-series data includes holdout data, and wherein adapting the predictive model to solve the prediction problem further comprises:

testing the first and second fitted models on the holdout data.

92. The apparatus of claim 61, wherein adapting the predictive model to solve the prediction problem further comprises performing nested cross-validation of the predictive model.

93. The apparatus of claim 92, wherein performing the nested cross-validation of the predictive model comprises:

partitioning the time-series data into a first plurality of partitions including at least a first partition of the time-series data and a second partition of the time-series data; and

partitioning the first partition of the time-series data into a plurality of partitions of the first partition of the time-series data including at least a first partition of the first partition of the time-series data and a second partition of the first partition of the time-series data,

wherein the training data comprise the first partition of the first partition of the time-series data, and wherein the testing data comprise at least a plurality of the partitions of the first partition of the time-series data other than the first partition of the first partition of the time-series data.

94. The apparatus of claim 93, wherein the training data are first training data, the testing data are first testing data, the fitted model is a first fitted model, and performing the nested cross-validation of the predictive model further comprises:

(i) generating, from the first partition of the time-series data, second training data and second testing data, wherein the second training data comprise the second partition of the first partition of the time-series data, and wherein the second testing data comprise at least a plurality of the partitions of the first partition of the data set other than the second partition of the first partition of the time-series data;

(j) fitting the predictive model to the second training data to obtain a second fitted model; and

(k) testing the second fitted model on the second testing data.

95. The apparatus of claim 94, wherein performing the nested cross-validation further comprises:

testing the first fitted model and the second fitted model on the second partition of the time-series data; and

comparing the first fitted model to the second fitted model based on results of testing the first and second fitted models on the second partition of the time-series data.

96. The apparatus of claim 61, wherein the memory is further configured to store instructions for deploying the fitted model, the processor is configured to execute the instructions for deploying the fitted model, and executing the instructions for deploying the fitted model causes the apparatus to deploy the fitted model, wherein the time-series data are first-time series data, and wherein deploying the fitted model includes:

obtaining second time-series data representing one or more instances of the prediction problem, wherein at least a portion of the second time-series data corresponds to a time period more recent than any portion of the first time-series data; and

generating at least one prediction associated with a future time by applying the fitted model to the second time-series data.

97. The apparatus of claim 96, wherein the memory is further configured to store instructions for refreshing the fitted model, the processor is configured to execute the instructions for refreshing the fitted model, and executing the instructions for refreshing the fitted model causes the apparatus to refresh the fitted model, including:

performing the predictive modeling procedure on third time-series data comprising at least a portion of the first time-series data and at least a portion of the second time-series data, thereby generating a refreshed model.

98. The apparatus of claim 61, wherein the variables include a first variable and a second variable, the memory is further configured to store analysis instructions, the processor is configured to execute the analysis instructions, and executing the analysis instructions causes the apparatus to perform acts including:

determining that changes in the values of the first and second variables are correlated, with a temporal lag between the changes in the value of the first variable and the correlated changes in the value of the second variable; and

displaying, via a graphical user interface, graphical content indicating a duration of the temporal lag between the changes in the value of the first variable and the correlated changes in the value of the second variable.

99. The apparatus of claim 61, wherein adapting the predictive model to solve the prediction problem further comprises down-sampling the training data prior to fitting the predictive model to the training data, and wherein down-sampling the training data comprises:

(1) removing, from the training data, all observations obtained from at least one of the data sets, or

(2) setting a value of a down-sampled time interval parameter of the training data to represent an integer multiple of the time interval parameter value of the time series data; and

for each instance of the down-sampled time interval parameter value of the training data;

identifying all observations in the training data associated with times corresponding to the respective instance of the down-sampled time interval parameter value of the training data,

aggregating the identified observations to generate an aggregate observation, and

replacing the identified observations in the training data with the aggregate observation.

100. The apparatus of claim 61, wherein adapting the predictive model to solve the prediction problem further comprises down-sampling the testing data prior to testing the fitted model on the testing data.

101. The apparatus of claim 61, wherein the memory is further configured to store evaluation instructions, the processor is configured to execute the evaluation instructions, and executing the evaluation instructions causes the apparatus to perform acts including:

determining suitabilities of a plurality of predictive modeling procedures for the prediction problem based, at least in part, on characteristics of the prediction problem and/or on attributes of the respective predictive modeling procedures;

selecting one or more predictive modeling procedures from the plurality of predictive modeling procedures based on the determined suitabilities of the selected modeling procedures for the prediction problem; and

performing the one or more predictive modeling procedures.

102. The apparatus of claim 101, wherein performing the one or more predictive modeling procedures comprises:

transmitting instructions to a plurality of processing nodes, the instructions comprising a resource allocation schedule allocating resources of the processing nodes for execution of the selected modeling procedures, the resource allocation schedule being based, at least in part, on the suitabilities of the selected modeling procedures for the prediction problem;

receiving results of the execution of the selected modeling procedures by the plurality of processing nodes in accordance with the resource allocation schedule, wherein the results include predictive models generated by the selected modeling procedures, and/or scores of the generated models for time-series data associated with the prediction problem; and

selecting, from the generated models, a predictive model for the prediction problem based, at least in part, on the score of the selected predictive model.

103. The apparatus of claim 61, wherein performing the predictive modeling procedure further comprises generating a blended predictive model by blending the fitted model with another fitted model.

104. The apparatus of claim 61, wherein the memory is further configured to store instructions for deploying the fitted model, the processor is configured to execute the instructions for deploying the fitted model, and executing the instructions for deploying the fitted model causes the apparatus to deploy the fitted model, wherein the time-series data are first time-series data, and wherein deploying the fitted model comprises refreshing the fitted model based, at least in part, on second time-series data.

105. The apparatus of claim 104, wherein the fitted model is a first fitted model, and wherein refreshing the fitted model based, at least in part, on the second time-series data comprises:

performing the predictive modeling procedure on the second time-series data to generate a second fitted model; and

blending the first fitted model and the second fitted model to generate a refreshed predictive model.

106. The apparatus of claim 61, wherein the memory is further configured to store instructions for deploying the fitted model, the processor is configured to execute the instructions for deploying the fitted model, and executing the instructions for deploying the fitted model causes the apparatus to deploy the fitted model, wherein the fitted model is deployed to one or more servers, wherein other fitted models are also deployed to the one or more servers, and wherein prediction requests to the fitted model and the other fitted models are allocated among the servers based, at least in part, on (1) an estimate of an amount of time used by each of the fitted models to generate a prediction, and/or (2) an estimate of a frequency with which prediction requests for each of the fitted models are received.

107. The apparatus of claim 106, wherein each prediction request is assigned to a respective thread, wherein each prediction request has an associated latency-sensitivity value, and wherein a number of threads executing on a particular server is determined based, at least in part, on the latency-sensitivity values of the threads executing on the particular server.

108. The apparatus of claim 61, wherein the memory is further configured to store analysis instructions, the processor is configured to execute the analysis instructions, and executing the analysis instructions causes the apparatus to perform acts including:

determining a value of a metric that indicates an interaction strength of two or more of the features included in the time-series data; and

if the value of the metric exceeds a threshold value, generating time-series values of a new feature based on the values of the two or more features and adding the new feature to the time-series data.

109. The apparatus of claim 61, wherein the memory is further configured to store analysis instructions, the processor is configured to execute the analysis instructions, and executing the analysis instructions causes the apparatus to perform acts including:

determining, for the fitted model, model-specific predictive values of one or more of the features of the time-series data; and

based at least in part on the model-specific predictive values of the features, performing at least one action selected from the group consisting of;

pruning a feature from the time-series data, creating a derived feature from two or more features in the time-series data and adding the derived feature to the time-series data, blending the predictive model with another predictive model, and/or allocating resources during a process of evaluating suitabilities of predictive modeling procedures for the prediction problem.