Dataset discovery in data analytics

• US 9,235,630 B1
• Filed: 09/25/2013
• Issued: 01/12/2016
• Est. Priority Date: 09/25/2013
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Dietrich, David, Todd, Stephen, Baldwin, Rhonda, Pauley, Wayne A. Jr.
• Primary Examiner(s): Holmes, Michael B
• Application Number:
  US14/036,612
• Time in Patent Office: 839 Days
• Field of Search: 706/45
• US Class Current: 1/1
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems

Abstract

An initial work package is obtained. The initial work package defines at least one hypothesis associated with a given data problem, and is generated in accordance with one or more phases of an automated data analytics lifecycle. A plurality of datasets is identified. One or more datasets in the plurality of datasets that are relevant to the at least one hypothesis are discovered. The at least one hypothesis is tested using at least a portion of the one or more discovered datasets.

Claims

1. A method comprising:

obtaining an initial work package defining at least one hypothesis associated with a given data problem, the initial work package being generated in accordance with one or more phases of an automated data analytics lifecycle;

identifying a plurality of datasets;

discovering one or more datasets in the plurality of datasets that are relevant to the at least one hypothesis, wherein discovery comprises performing data mining on the plurality of datasets; and

testing the at least one hypothesis using at least a portion of the one or more discovered datasets;

wherein the obtaining, identifying, discovering, and testing steps are performed on one or more processing elements associated with a computing system.

2. The method of claim 1, wherein the step of data mining of the plurality of datasets further comprises identifying one or more datasets in the plurality of datasets based on the initial work package.

3. The method of claim 2, wherein the step of discovering one or more datasets in the plurality of datasets that are relevant to the at least one hypothesis further comprises generating one or more data catalogs from the one or more identified datasets.

4. The method of claim 3, wherein the step of discovering one or more datasets in the plurality of datasets that are relevant to the at least one hypothesis further comprises identifying associations between at least a portion of the one or more datasets in the one or more data catalogs.

5. The method of claim 4, wherein the step of discovering one or more datasets in the plurality of datasets that are relevant to the at least one hypothesis further comprises generating provenance data for at least a portion of the one or more datasets in the one or more data catalogs.

6. The method of claim 2, wherein the step of discovering one or more datasets in the plurality of datasets that are relevant to the at least one hypothesis further comprises extracting one or more features from the one or more identified datasets.

7. The method of claim 6, wherein extracting one or more features from the one or more datasets further comprises determining the presence of private data in the one or more identified datasets.

8. The method of claim 7, further comprising preventing use of the private data from the one or more identified datasets.

9. The method of claim 7, further comprising masking the private data in the one or more identified datasets.

10. The method of claim 6, wherein extracting one or more features from the one or more datasets further comprises determining one or more properties associated with the one or more identified datasets.

11. The method of claim 10, wherein the one or more properties comprise at least one of a data change rate, a data structure type, and data use type.

12. The method of claim 1, further comprising refining the at least one hypothesis as a result of the testing step.

13. The method of claim 12, further comprising updating the initial work package based on the refined hypothesis to generate a refined work package for use by the one or more phases of the automated data analytics lifecycle.

14. A computer program product comprising a processor-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by one or more processing elements of the computing system implement the steps of:

obtaining an initial work package defining at least one hypothesis associated with a given data problem, the initial work package being generated in accordance with one or more phases of an automated data analytics lifecycle;

identifying a plurality of datasets;

discovering one or more datasets in the plurality of datasets that are relevant to the at least one hypothesis, wherein discovery comprises performing data mining on the plurality of datasets; and

testing the at least one hypothesis using at least a portion of the one or more discovered datasets.

15. An apparatus comprising:

a memory; and

at least one processor operatively coupled to the memory and configured to;

obtain an initial work package defining at least one hypothesis associated with a given data problem, the initial work package being generated in accordance with one or more phases of an automated data analytics lifecycle;

identify a plurality of datasets;

discover one or more datasets in the plurality of datasets that are relevant to the at least one hypothesis, wherein discovery comprises performing data mining on the plurality of datasets; and

test the at least one hypothesis using at least a portion of the one or more discovered datasets.

16. The apparatus of claim 15, wherein data mining of the plurality of datasets further comprises identifying one or more datasets in the plurality of datasets based on the initial work package.

17. The apparatus of claim 16, wherein discovering one or more datasets in the plurality of datasets that are relevant to the at least one hypothesis further comprises generating one or more data catalogs from the one or more identified datasets.

18. The apparatus of claim 16, wherein the step of discovering one or more datasets in the plurality of datasets that are relevant to the at least one hypothesis further comprises extracting one or more features from the one or more identified datasets.

19. The apparatus of claim 15, wherein the at least one processor is further configured to refine the at least one hypothesis as a result of the testing step.

20. The apparatus of claim 19, wherein the at least one processor is further configured to update the initial work package based on the refined hypothesis to generate a refined work package for use by the one or more phases of the automated data analytics lifecycle.