Data analytics lifecycle processes

• US 9,262,493 B1
• Filed: 12/27/2012
• Issued: 02/16/2016
• Est. Priority Date: 12/27/2012
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Dietrich, David
• Primary Examiner(s): AL HASHEMI, SANA A
• Application Number:
  US13/728,394
• Time in Patent Office: 1,146 Days
• Field of Search: 707/608, 707/609, 707/687, 707/705, 707/790, 707/821
• US Class Current: 1/1
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

A data analytic plan is defined for analyzing a given data set associated with a given data problem. A test data set and a training data set are obtained from the given data set associated with the given data problem. At least one model is executed to confirm an adequacy of the at least one model for the data analytic plan by fitting the at least one model on the training data set and evaluating the at least one model fitted on the training data set against the test data set. The defining, obtaining and executing steps are performed on one or more processing elements associated with a computing system and automate at least part of a data analytics lifecycle.

Claims

1. A method comprising:

defining a data analytic plan for analyzing a given data set associated with a given data problem associated with a data analytics lifecycle;

obtaining a test data set and a training data set from the given data set associated with the given data problem;

executing at least one model to confirm an adequacy of the at least one model for the data analytic plan by fitting the at least one model on the training data set and evaluating the at least one model fitted on the training data set against the test data set, wherein the evaluation comprises assessing a validity of the at least one model and a validity of results of the execution of the at least one model on the test data set;

refining the at least one model based on the assessment;

conditioning at least a portion of raw data in the given data set to generate conditioned data;

creating an analytics environment in which the executing, evaluating and conditioning steps are performed, the analytics environment comprising parameters including at least a capacity and a bandwidth of the analytics environment; and

dynamically changing the parameters in response to the refining step to include parameters to perform additional executing and evaluating steps on refinements of the at least one model;

wherein the step of dynamically changing the parameters is performed such that the data analytics lifecycle is configured to continue from a point in the lifecycle where the parameters were changed;

wherein the execution of the at least one model is performed prior to implementation of the data analytic plan in a destination environment;

wherein the training data set is used to train the at least one model and the test data set is used to determine the accuracy of the at least one model fitted on the training data set; and

wherein the defining, obtaining, executing, refining, conditioning, creating and dynamically changing steps are performed on one or more processing elements associated with a computing system and automate at least part of the data analytics lifecycle.

4. The method of claim 1, wherein the model execution step further comprises converting the model to a database language prior to execution.

5. The method of claim 1, wherein the step of refining the at least one model further comprises modifying one or more model variables.

6. The method of claim 1, wherein the test data set is smaller than the training data set.

7. The method of claim 1, wherein the data analytic plan definition step further comprises determining at least one category of problem type based on one or more analytical methods.

8. The method of claim 1, wherein the data analytic plan definition step further comprises assessing available resources based on the data analytic plan.

9. The method of claim 1, wherein the data analytic plan definition step further comprises identifying data sources based on the data analytic plan.

10. The method of claim 1, wherein the raw data conditioning step further comprises performing an extract, load and transform process on the portion of the raw data.

11. The method of claim 1, wherein the raw data conditioning step further comprises performing an extract, transform, load and transform process on the portion of the raw data.

12. The method of claim 1, further comprising interpreting one or more results of the model execution step by comparing the one or more results to the data analytic plan.

13. The method of claim 1, further comprising interpreting one or more results of the model execution step by measuring a business value of the one or more results.

14. The method of claim 1, further comprising deploying the data analytic plan in a controlled subset of the destination environment in which the data analytic plan could otherwise be fully deployed.

15. A computer program product comprising a processor-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by the one or more processing elements of the computing system implement the steps of the method of claim 1.

16. The method of claim 4, wherein the model execution step further comprises executing the database language-converted model as one or more functions within the database.

17. The method of claim 14, further comprising retraining the at least one model after deploying the data analytic plan in the controlled subset of the destination environment.

18. The method of claim 17, further comprising re-executing the at least one model after the retraining step.

2. An apparatus comprising:

a memory; and

at least one processor operatively coupled to the memory and configured to;

define a data analytic plan for analyzing a given data set associated with a given data problem associated with a data analytics lifecycle;

obtain a test data set and a training data set from the given data set associated with the given data problem;

execute the at least one model to confirm an adequacy of the at least one model for the data analytic plan by fitting the at least one model on the training data set and evaluate the at least one model fitted on the training data set against the test data set, wherein the evaluation comprises assessing a validity of the at least one model and a validity of results of the execution of the at least one model on the test data set;

refine the at least one model based on the assessment;

condition at least a portion of raw data in the given data set to generate conditioned data;

create an analytics environment in which the executing, evaluating and conditioning operations are performed, the analytics environment comprising parameters including at least a capacity and a bandwidth of the analytics environment; and

dynamically change the parameters in response to the refining operation to include parameters to perform additional executing and evaluating operations on refinements of the at least one model;

wherein the operation of dynamically changing the parameters is performed such that the data analytics lifecycle is configured to continue from a point in the lifecycle where the parameters were changed;

wherein the execution of the at least one model is performed prior to implementation of the data analytic plan in a destination environment;

wherein the training data set is used to train the at least one model and the test data set is used to determine the accuracy of the at least one model fitted on the training data set; and

wherein the defining, obtaining, executing, refining, conditioning, creating and dynamically changing operations automate at least part of the data analytics lifecycle.

3. A system comprising:

a discovery phase module for defining a data analytic plan for analyzing a given data set associated with a given data problem associated with a data analytics lifecycle;

a model building phase module for obtaining a test data set and a training data set from the given data set associated with the given data problem, executing the at least one model to confirm an adequacy of the at least one model for the data analytic plan by fitting the at least one model on the training data set and evaluating the at least one model fitted on the training data set against the test data set, wherein the evaluation comprises assessing a validity of the at least one model and a validity of results of the execution of the at least one model on the test data set, refining the at least one model based on the assessment; and

a data preparation phase module for conditioning at least a portion of raw data in the given data set to generate conditioned data, and for creating an analytics environment in which the executing, evaluating and conditioning operations are performed, the analytics environment comprising parameters including at least a capacity and a bandwidth of the analytics environment;

wherein the parameters are dynamically changed in response to the refining operation to include parameters to perform additional executing and evaluating operations on refinements of the at least one model;

wherein the operation of dynamically changing the parameters is performed such that the data analytics lifecycle is configured to continue from a point in the lifecycle where the parameters were changed;

wherein the execution of the at least one model is performed prior to implementation of the data analytic plan in a destination environment;

wherein the training data set is used to train the at least one model and the test data set is used to determine the accuracy of the at least one model fitted on the training data set; and

wherein the discovery phase module, the model building phase module, and the data preparation phase module are implemented on one or more processing elements associated with a computing system so as to automate at least part of the data analytics lifecycle.