Data profiling

• US 20050114369A1
• Filed: 09/15/2004
• Published: 05/26/2005
• Est. Priority Date: 09/15/2003
• Status: Active Grant

• Current Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Original Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Inventors: Gould, Joel, Feynman, Carl, Bay, Paul
• Granted Patent: US 8,868,580 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24544 Join order optimisation
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/285 Clustering or classification

Abstract

Processing data includes profiling data from a data source, including reading the data from the data source, computing summary data characterizing the data while reading the data, and storing profile information that is based on the summary data. The data is then processed from the data source. This processing includes accessing the stored profile information and processing the data according to the accessed profile information.

Claims

1. A method for processing data including:

profiling data from a data source, including reading the data from the data source, computing summary data characterizing the data while reading the data, and storing profile information that is based on the summary data; and

processing the data from the data source, including accessing the stored profile information and processing the data according to the accessed profile information.

2. The method of claim 1 wherein processing the data from the data source further includes reading the data from the data source.

3. The method of claim 1 wherein profiling the data is performed without maintaining a copy of the data outside the data source.

4. The method of claim 3 wherein the data includes variable record structure records with at least one of a conditional field and a variable number of fields.

5. The method of claim 4 wherein computing summary data characterizing the data while reading the data includes interpreting the variable record structure records while computing summary data characterizing the data.

6. The method of claim 1 wherein the data source includes a data storage system.

7. The method of claim 6 wherein the data storage system includes a database system.

8. The method of claim 1 wherein computing the summary data includes counting a number of occurrences for each of a set of distinct values for a field.

9. The method of claim 8 wherein storing profile information includes storing statistics for the field based on the counted number of occurrences for said field.

10. The method of claim 1 further including maintaining a metadata store that contains metadata related to the data source

11. The method of claim 10 wherein storing the profile information includes updating the metadata related to the data source.

12. The method of claim 10 wherein profiling the data and processing the data each make use of metadata for the data source

13. The method of claim 1 wherein profiling data from the data source further includes determining a format specification based on the profile information.

14. The method of claim 1 wherein profiling data from the data source further includes determining a validation specification based on the profile information.

15. The method of claim 14 wherein processing the data includes identifying invalid records in the data based on the validation specification.

16. The method of claim 1 wherein profiling data from the data source further includes specifying data transformation instructions based on the profile information.

17. The method of claim 16 wherein processing the data includes applying the transformation instructions to the data.

18. The method of claim 1 wherein processing the data includes importing the data into a data storage subsystem.

19. The method of claim 18 wherein processing the data includes validating the data prior to importing the data into a data storage subsystem.

20. The method of claim 19 wherein validating the data includes comparing characteristics of the data to reference characteristics for said data.

21. The method of claim 20 wherein the reference characteristics include statistical properties of the data.

22. The method of claim 1 wherein profiling the data includes profiling said data in parallel, including partitioning the data into parts and processing the parts using separate ones of a first set of parallel components.

23. The method of claim 22 wherein profiling the data in parallel further includes computing the summary data for different fields of the data using separate ones of a second set of parallel components.

24. The method of claim 23 wherein profiling the data in parallel further includes repartitioning outputs of the first set of parallel components to form inputs for the second set of parallel components.

25. The method of claim 22 wherein profiling the data in parallel includes reading the data from a parallel data source, each part of the parallel data source being processed by a different one of the first set of parallel components.

26. A method for processing data including:

profiling data from a data source, including reading the data from the data source, computing summary data characterizing the data while reading the data, and storing profile information that is based on the summary data;

wherein

profiling the data includes profiling said data in parallel, including partitioning the data into parts and processing the parts using separate ones of a first set of parallel components.

27. Software stored on a computer-readable medium including instructions for causing a computer system to:

profile data from a data source by reading the data from the data source, compute summary data characterizing the data while reading the data, and store profile information that is based on the summary data; and

process the data from the data source by accessing the stored profile information and process the data according to the accessed profile information.

28. A data processing system including:

a profiling module configured to read data from a data source, to compute summary data characterizing the data while reading the data, and to store profile information that is based on the summary data; and

a processing module configured to access the stored profile information and to process the data from the data source according to the accessed profile information.

29. A data processing system including:

means for profiling data from a data source, including means for reading the data from the data source, means for computing summary data characterizing the data while reading the data, and means for storing profile information that is based on the summary data; and

means for processing the data from the data source, including means for accessing the stored profile information and means for processing the data according to the accessed profile information.