Profiling data with source tracking

• US 10,719,511 B2
• Filed: 02/13/2017
• Issued: 07/21/2020
• Est. Priority Date: 10/22/2012
• Status: Active Grant

• Current Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Original Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Inventors: Anderson, Arlen
• Primary Examiner(s): Ly, Anh
• Application Number:
  US15/431,008
• Publication Number:
  US 20170154075A1
• Time in Patent Office: 1,254 Days
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24544 Join order optimisation
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database

Abstract

Profiling data includes accessing multiple collections of records to store quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, each including a value appearing in the selected field and a count of the number of records in which the value appears. Processing the quantitative information of two or more collections includes: merging the value count entries of corresponding lists for at least one field from each of a first collection and a second collection to generate a combined list of value count entries, and aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries identifying a distinct value and including information quantifying a number of records in which the distinct value appears for each of the two or more collections.

Claims

1. A method for profiling data stored in a data storage system, the method including:

accessing multiple collections of records stored in the data storage system over an interface coupled to the data storage system;

determining quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in at least the selected field and a count of the number of records in which the value appears in at least the selected field; and

processing the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

merging the value count entries of corresponding lists for at least one field from each of at least a first collection and a second collection of the two or more collections to generate a combined list of value count entries, and

aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries, at least some of the distinct field value entries identifying a distinct value from at least one of the value count entries;

wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in at least the selected field.

2. The method of claim 1, wherein merging the value count entries of the corresponding lists includes sorting the value count entries based on the values from the value count entries.

3. The method of claim 2, wherein sorting the value count entries includes sorting by a primary sort order based on the values from the value count entries, and sorting by a secondary sort order based on identifiers of the two or more collections.

4. The method of claim 1, wherein processing the quantitative information of two or more of the collections includes processing the quantitative information of three or more of the collections.

5. The method of claim 4, wherein the processing includes, for a first subset of at least two of the three or more collections, generating profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the first subset.

6. The method of claim 5, wherein the processing includes, for a second subset of at least two of the three or more collections, different from the first subset, generating profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the second subset.

7. The method of claim 1, wherein the processing includes:

reading the value count entries of a corresponding list for at least one field from a third collection of the two or more collections to update the list of distinct field value entries so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first, second, and third collections and include information quantifying a number of records in which the distinct value appears for each of the first, second, and third collections.

8. A computer program, stored on a computer-readable storage medium, for profiling data stored in a data storage system, the computer program including instructions for causing a computing system to:

access multiple collections of records stored in the data storage system over an interface coupled to the data storage system;

determining quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in at least the selected field and a count of the number of records in which the value appears in at least the selected field; and

process the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

merging the value count entries of corresponding lists for at least one field from each of at least a first collection and a second collection of the two or more collections to generate a combined list of value count entries, and

aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries, at least some of the distinct field value entries identifying a distinct value from at least one of the value count entries;

wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in at least the selected field.

9. The computer program of claim 8, wherein merging the value count entries of the corresponding lists includes sorting the value count entries based on the values from the value count entries.

10. The computer program of claim 9, wherein sorting the value count entries includes sorting by a primary sort order based on the values from the value count entries, and sorting by a secondary sort order based on identifiers of the two or more collections.

11. The computer program of claim 8, wherein processing the quantitative information of two or more of the collections includes processing the quantitative information of three or more of the collections.

12. The computer program of claim 11, wherein the processing includes, for a first subset of at least two of the three or more collections, generating profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the first subset.

13. The computer program of claim 12, wherein the processing includes, for a second subset of at least two of the three or more collections, different from the first subset, generating profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the second subset.

14. The computer program of claim 8, wherein the processing includes:

reading the value count entries of a corresponding list for at least one field from a third collection of the two or more collections to update the list of distinct field value entries so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first, second, and third collections and include information quantifying a number of records in which the distinct value appears for each of the first, second, and third collections.

15. A computing system for profiling data stored in a data storage system, the computing system including:

an interface coupled to the data storage system configured to access multiple collections of records stored in the data storage system;

determining quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in at least the selected field and a count of the number of records in which the value appears in at least the selected field; and

at least one processor configured to process the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

merging the value count entries of corresponding lists for at least one field from each of at least a first collection and a second collection of the two or more collections to generate a combined list of value count entries, and

aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries, at least some of the distinct field value entries identifying a distinct value from at least one of the value count entries;

wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in at least the selected field.

16. The computing system of claim 15, wherein merging the value count entries of the corresponding lists includes sorting the value count entries based on the values from the value count entries.

17. The computing system of claim 16, wherein sorting the value count entries includes sorting by a primary sort order based on the values from the value count entries, and sorting by a secondary sort order based on identifiers of the two or more collections.

18. The computing system of claim 15, wherein processing the quantitative information of two or more of the collections includes processing the quantitative information of three or more of the collections.

19. The computing system of claim 18, wherein the processing includes, for a first subset of at least two of the three or more collections, generating profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the first subset.

20. The computing system of claim 19, wherein the processing includes, for a second subset of at least two of the three or more collections, different from the first subset, generating profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the second subset.

21. The computing system of claim 15, wherein the processing includes:

reading the value count entries of a corresponding list for at least one field from a third collection of the two or more collections to update the list of distinct field value entries so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first, second, and third collections and include information quantifying a number of records in which the distinct value appears for each of the first, second, and third collections.

22. A computing system for profiling data stored in a data storage system, the computing system including:

means for accessing multiple collections of records stored in the data storage system;

means for determining quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in at least the selected field and a count of the number of records in which the value appears in at least the selected field; and

means for processing the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

merging the value count entries of corresponding lists for at least one field from each of at least a first collection and a second collection of the two or more collections to generate a combined list of value count entries, and

aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries, at least some of the distinct field value entries identifying a distinct value from at least one of the value count entries;

wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in at least the selected field.

23. A method for profiling data stored in a data storage system, the method including:

accessing multiple collections of records stored in the data storage system over an interface coupled to the data storage system;

and

processing quantitative information of two or more of the collections to generate profiling summary information, the processing including;

determining the quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in at least the selected field and a count of the number of records in which the value appears in at least the selected field,

reading the value count entries of a corresponding list for at least one field from a first collection of the two or more collections to store output data that includes a list of distinct field value entries, and

reading the value count entries of a corresponding list for at least one field from a second collection of the two or more collections to store updated output data based at least in part on the stored output data so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first and second collections;

wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in at least the selected field.

24. The method of claim 23, wherein processing the quantitative information of two or more of the collections includes processing the quantitative information of three or more of the collections.

25. The method of claim 24, further including, for a first subset of at least two of the three or more collections, generating profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the first subset.

26. The method of claim 25, further including, for a second subset of at least two of the three or more collections, different from the first subset, generating profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the second subset.

27. The method of claim 23, wherein the processing includes:

reading the value count entries of a corresponding list for at least one field from a third collection of the two or more collections to update the list of distinct field value entries so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first, second, and third collections and include information quantifying a number of records in which the distinct value appears for each of the first, second, and third collections.

28. A computer program, stored on a computer-readable storage medium, for profiling data stored in a data storage system, the computer program including instructions for causing a computing system to:

access multiple collections of records stored in the data storage system over an interface coupled to the data storage system;

determine quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in at least the selected field and a count of the number of records in which the value appears in at least the selected field; and

process the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

reading the value count entries of a corresponding list for at least one field from a first collection of the two or more collections to store output data that includes a list of distinct field value entries, and

reading the value count entries of a corresponding list for at least one field from a second collection of the two or more collections to store updated output data based at least in part on the stored output data so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first and second collections;

wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in at least the selected field.

29. The computer program of claim 28, wherein processing the quantitative information of two or more of the collections includes processing the quantitative information of three or more of the collections.

30. The computer program of claim 29, wherein the processing includes, for a first subset of at least two of the three or more collections, generating profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the first subset.

31. The computer program of claim 30, wherein the processing includes, for a second subset of at least two of the three or more collections, different from the first subset, generating profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the second subset.

32. The computer program of claim 28, wherein the processing includes:

reading the value count entries of a corresponding list for at least one field from a third collection of the two or more collections to update the list of distinct field value entries so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first, second, and third collections and include information quantifying a number of records in which the distinct value appears for each of the first, second, and third collections.

33. A computing system for profiling data stored in a data storage system, the computing system including:

an interface coupled to the data storage system configured to access multiple collections of records stored in the data storage system;

and

at least one processor configured to process quantitative information of two or more of the collections to generate profiling summary information, the processing including;

determining the quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in at least the selected field and a count of the number of records in which the value appears in at least the selected field,

reading the value count entries of a corresponding list for at least one field from a first collection of the two or more collections to store output data that includes a list of distinct field value entries, and

reading the value count entries of a corresponding list for at least one field from a second collection of the two or more collections to store updated output data based at least in part on the stored output data so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first and second collections;

wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in at least the selected field.

34. The computing system of claim 33, wherein processing the quantitative information of two or more of the collections includes processing the quantitative information of three or more of the collections.

35. The computing system of claim 34, wherein the processing includes, for a first subset of at least two of the three or more collections, generating profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the first subset.

36. The computing system of claim 35, wherein the processing includes, for a second subset of at least two of the three or more collections, different from the first subset, generating profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the second subset.

37. The computing system of claim 33, wherein the processing includes:

reading the value count entries of a corresponding list for at least one field from a third collection of the two or more collections to update the list of distinct field value entries so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first, second, and third collections and include information quantifying a number of records in which the distinct value appears for each of the first, second, and third collections.

38. A computing system for profiling data stored in a data storage system, the computing system including:

means for accessing multiple collections of records stored in the data storage system;

means for determining quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in at least the selected field and a count of the number of records in which the value appears in at least the selected field; and

means for processing the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

reading the value count entries of a corresponding list for at least one field from a first collection of the two or more collections to store output data that includes a list of distinct field value entries, and

reading the value count entries of a corresponding list for at least one field from a second collection of the two or more collections to store updated output data based at least in part on the stored output data so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first and second collections;

wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in at least the selected field.