Profiling data with source tracking

• US 9,569,434 B2
• Filed: 08/02/2013
• Issued: 02/14/2017
• Est. Priority Date: 10/22/2012
• Status: Active Grant

• Current Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Original Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Inventors: Anderson, Arlen
• Primary Examiner(s): Ly, Anh
• Application Number:
  US13/957,641
• Publication Number:
  US 20140114926A1
• Time in Patent Office: 1,292 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24544 Join order optimisation
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database

Abstract

Profiling data includes accessing multiple collections of records to store quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, each including a value appearing in the selected field and a count of the number of records in which the value appears. Processing the quantitative information of two or more collections includes: merging the value count entries of corresponding lists for at least one field from each of a first collection and a second collection to generate a combined list of value count entries, and aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries identifying a distinct value and including information quantifying a number of records in which the distinct value appears for each of the two or more collections.

Claims

1. A method for profiling data stored in a data storage system, the method including:

accessing multiple collections of records stored in the data storage system over an interface coupled to the data storage system to store quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in the selected field and a count of the number of records in which the value appears in the selected field; and

processing the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

merging the value count entries of corresponding lists for at least one field from each of at least a first collection and a second collection of the two or more collections to generate a combined list of value count entries, and

aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries, at least some of the distinct field value entries identifying a distinct value from at least one of the value count entries and including information quantifying a number of records in which the distinct value appears for each of the two or more collections;

wherein processing the quantitative information of two or more of the collections includes processing the quantitative information of three or more of the collections; and

the method further including;

for a first subset of at least two of the three or more collections, generating profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the first subset; and

for a second subset of at least two of the three or more collections, different from the first subset, generating profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the second subset.

2. The method of claim 1, wherein merging the value count entries of the corresponding lists includes sorting the value count entries based on the values from the value count entries.

3. The method of claim 2, wherein sorting the value count entries includes sorting by a primary sort order based on the values from the value count entries, and sorting by a secondary sort order based on identifiers of the two or more collections.

4. The method of claim 1, wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in the selected field.

5. The method of claim 1, wherein the processing includes:

reading the value count entries of a corresponding list for at least one field from a third collection of the two or more collections to update the list of distinct field value entries so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first, second, and third collections and include information quantifying a number of records in which the distinct value appears for each of the first, second, and third collections.

6. A computer program, stored on a computer-readable storage medium, for profiling data stored in a data storage system, the computer program including instructions for causing a computing system to:

access multiple collections of records stored in the data storage system over an interface coupled to the data storage system to store quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in the selected field and a count of the number of records in which the value appears in the selected field; and

process the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

merging the value count entries of corresponding lists for at least one field from each of at least a first collection and a second collection of the two or more collections to generate a combined list of value count entries, and

aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries, at least some of the distinct field value entries identifying a distinct value from at least one of the value count entries and including information quantifying a number of records in which the distinct value appears for each of the two or more collections;

wherein processing the quantitative information of two or more of the collections includes processing the quantitative information of three or more of the collections; and

the computer program further including instructions for causing a computing system to;

for a first subset of at least two of the three or more collections, generate profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the first subset; and

for a second subset of at least two of the three or more collections, different from the first subset, generate profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the second subset.

7. The computer program of claim 6, wherein merging the value count entries of the corresponding lists includes sorting the value count entries based on the values from the value count entries.

8. The computer program of claim 7, wherein sorting the value count entries includes sorting by a primary sort order based on the values from the value count entries, and sorting by a secondary sort order based on identifiers of the two or more collections.

9. The computer program of claim 6, wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in the selected field.

10. The computer program of claim 6, wherein the processing includes:

reading the value count entries of a corresponding list for at least one field from a third collection of the two or more collections to update the list of distinct field value entries so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first, second, and third collections and include information quantifying a number of records in which the distinct value appears for each of the first, second, and third collections.

11. A computing system for profiling data stored in a data storage system, the computing system including:

an interface coupled to the data storage system configured to access multiple collections of records stored in the data storage system to store quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in the selected field and a count of the number of records in which the value appears in the selected field; and

at least one processor configured to process the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

merging the value count entries of corresponding lists for at least one field from each of at least a first collection and a second collection of the two or more collections to generate a combined list of value count entries, and

aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries, at least some of the distinct field value entries identifying a distinct value from at least one of the value count entries and including information quantifying a number of records in which the distinct value appears for each of the two or more collections;

wherein processing the quantitative information of two or more of the collections includes processing the quantitative information of three or more of the collections; and

the at least one processor further configured to;

for a first subset of at least two of the three or more collections, generate profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the first subset; and

for a second subset of at least two of the three or more collections, different from the first subset, generate profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the second subset.

12. The computing system of claim 11, wherein merging the value count entries of the corresponding lists includes sorting the value count entries based on the values from the value count entries.

13. The computing system of claim 12, wherein sorting the value count entries includes sorting by a primary sort order based on the values from the value count entries, and sorting by a secondary sort order based on identifiers of the two or more collections.

14. The computing system of claim 11, wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in the selected field.

15. The computing system of claim 11, wherein the processing includes:

reading the value count entries of a corresponding list for at least one field from a third collection of the two or more collections to update the list of distinct field value entries so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first, second, and third collections and include information quantifying a number of records in which the distinct value appears for each of the first, second, and third collections.

16. A computing system for profiling data stored in a data storage system, the computing system including:

means for accessing multiple collections of records stored in the data storage system to store quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in the selected field and a count of the number of records in which the value appears in the selected field; and

means for processing the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

merging the value count entries of corresponding lists for at least one field from each of at least a first collection and a second collection of the two or more collections to generate a combined list of value count entries, and

aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries, at least some of the distinct field value entries identifying a distinct value from at least one of the value count entries and including information quantifying a number of records in which the distinct value appears for each of the two or more collections;

wherein processing the quantitative information of two or more of the collections includes processing the quantitative information of three or more of the collections; and

the processing further including;

for a first subset of at least two of the three or more collections, generate profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the first subset; and

for a second subset of at least two of the three or more collections, different from the first subset, generate profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the second subset.

17. A method for profiling data stored in a data storage system, the method including:

accessing multiple collections of records stored in the data storage system over an interface coupled to the data storage system to store quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in the selected field and a count of the number of records in which the value appears in the selected field; and

processing the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

reading the value count entries of a corresponding list for at least one field from a first collection of the two or more collections to store output data that includes a list of distinct field value entries, and

reading the value count entries of a corresponding list for at least one field from a second collection of the two or more collections to store updated output data based at least in part on the stored output data so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first and second collections and include information quantifying a number of records in which the distinct value appears for each of the first and second collection;

wherein processing the quantitative information of two or more of the collections includes processing the quantitative information of three or more of the collections and the method further including;

for a first subset of at least two of the three or more collections, generating profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the first subset; and

for a second subset of at least two of the three or more collections, different from the first subset, generating profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the second subset.

18. The method of claim 17, wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in the selected field.

19. The method of claim 17, wherein the processing includes:

reading the value count entries of a corresponding list for at least one field from a third collection of the two or more collections to update the list of distinct field value entries so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first, second, and third collections and include information quantifying a number of records in which the distinct value appears for each of the first, second, and third collections.

20. A computer program, stored on a computer-readable storage medium, for profiling data stored in a data storage system, the computer program including instructions for causing a computing system to:

access multiple collections of records stored in the data storage system over an interface coupled to the data storage system to store quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in the selected field and a count of the number of records in which the value appears in the selected field; and

process the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

reading the value count entries of a corresponding list for at least one field from a first collection of the two or more collections to store output data that includes a list of distinct field value entries, and

reading the value count entries of a corresponding list for at least one field from a second collection of the two or more collections to store updated output data based at least in part on the stored output data so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first and second collections and include information quantifying a number of records in which the distinct value appears for each of the first and second collections;

wherein processing the quantitative information of two or more of the collections includes processing the quantitative information of three or more of the collections; and

the computer program further including instructions for causing a computing system to;

for a first subset of at least two of the three or more collections, generate profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the first subset; and

for a second subset of at least two of the three or more collections, different from the first subset, generate profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the second subset.

21. The computer program of claim 20, wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in the selected field.

22. The computer program of claim 20, wherein the processing includes:

reading the value count entries of a corresponding list for at least one field from a third collection of the two or more collections to update the list of distinct field value entries so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first, second, and third collections and include information quantifying a number of records in which the distinct value appears for each of the first, second, and third collections.

23. A computing system for profiling data stored in a data storage system, the computing system including:

an interface coupled to the data storage system configured to access multiple collections of records stored in the data storage system to store quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in the selected field and a count of the number of records in which the value appears in the selected field; and

at least one processor configured to process the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

reading the value count entries of a corresponding list for at least one field from a first collection of the two or more collections to store output data that includes a list of distinct field value entries, and

reading the value count entries of a corresponding list for at least one field from a second collection of the two or more collections to store updated output data based at least in part on the stored output data so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first and second collections and include information quantifying a number of records in which the distinct value appears for each of the first and second collections;

wherein processing the quantitative information of two or more of the collections includes processing the quantitative information of three or more of the collections; and

the at least one processor further configured to;

for a first subset of at least two of the three or more collections, generate profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the first subset; and

for a second subset of at least two of the three or more collections, different from the first subset, generate profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the second subset.

24. The computing system of claim 23, wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in the selected field.

25. The computing system of claim 23, wherein the processing includes:

reading the value count entries of a corresponding list for at least one field from a third collection of the two or more collections to update the list of distinct field value entries so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first, second, and third collections and include information quantifying a number of records in which the distinct value appears for each of the first, second, and third collections.

26. A computing system for profiling data stored in a data storage system, the computing system including:

means for accessing multiple collections of records stored in the data storage system to store quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in the selected field and a count of the number of records in which the value appears in the selected field; and

means for processing the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

reading the value count entries of a corresponding list for at least one field from a first collection of the two or more collections to store output data that includes a list of distinct field value entries, and

reading the value count entries of a corresponding list for at least one field from a second collection of the two or more collections to store updated output data based at least in part on the stored output data so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first and second collections and include information quantifying a number of records in which the distinct value appears for each of the first and second collections;

wherein processing the quantitative information of two or more of the collections includes processing the quantitative information of three or more of the collections; and

the processing further including;

for a first subset of at least two of the three or more collections, generate profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a loin operation between the fields of respective collections of records in the first subset; and

for a second subset of at least two of the three or more collections, different from the first subset, generate profiling summary information from the list of distinct field value entries, the profiling summary information including multiple patterns of results of a join operation between the fields of respective collections of records in the second subset.

27. A method for profiling data stored in a data storage system, the method including:

accessing multiple collections of records stored in the data storage system over an interface coupled to the data storage system to store quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in the selected field and a count of the number of records in which the value appears in the selected field; and

processing the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

merging the value count entries of corresponding lists for at least one field from each of at least a first collection and a second collection of the two or more collections to generate a combined list of value count entries, and

aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries, at least some of the distinct field value entries identifying a distinct value from at least one of the value count entries and including information quantifying a number of records in which the distinct value appears for each of the two or more collections;

wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in the selected field.

28. The method of claim 27, wherein merging the value count entries of the corresponding lists includes sorting the value count entries based on the values from the value count entries.

29. The method of claim 28, wherein sorting the value count entries includes sorting by a primary sort order based on the values from the value count entries, and sorting by a secondary sort order based on identifiers of the two or more collections.

30. A computer program, stored on a computer-readable storage medium, for profiling data stored in a data storage system, the computer program including instructions for causing a computing system to:

access multiple collections of records stored in the data storage system over an interface coupled to the data storage system to store quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in the selected field and a count of the number of records in which the value appears in the selected field; and

process the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

merging the value count entries of corresponding lists for at least one field from each of at least a first collection and a second collection of the two or more collections to generate a combined list of value count entries, and

aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries, at least some of the distinct field value entries identifying a distinct value from at least one of the value count entries and including information quantifying a number of records in which the distinct value appears for each of the two or more collections;

wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in the selected field.

31. The computing system of claim 30, wherein merging the value count entries of the corresponding lists includes sorting the value count entries based on the values from the value count entries.

32. The computing system of claim 31, wherein sorting the value count entries includes sorting by a primary sort order based on the values from the value count entries, and sorting by a secondary sort order based on identifiers of the two or more collections.

33. A computing system for profiling data stored in a data storage system, the computing system including:

an interface coupled to the data storage system configured to access multiple collections of records stored in the data storage system to store quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in the selected field and a count of the number of records in which the value appears in the selected field; and

at least one processor configured to process the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

merging the value count entries of corresponding lists for at least one field from each of at least a first collection and a second collection of the two or more collections to generate a combined list of value count entries, and

aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries, at least some of the distinct field value entries identifying a distinct value from at least one of the value count entries and including information quantifying a number of records in which the distinct value appears for each of the two or more collections;

wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in the selected field.

34. The computing system of claim 33, wherein merging the value count entries of the corresponding lists includes sorting the value count entries based on the values from the value count entries.

35. The computing system of claim 34, wherein sorting the value count entries includes sorting by a primary sort order based on the values from the value count entries, and sorting by a secondary sort order based on identifiers of the two or more collections.

36. A computing system for profiling data stored in a data storage system, the computing system including:

means for accessing multiple collections of records stored in the data storage system to store quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in the selected field and a count of the number of records in which the value appears in the selected field; and

means for processing the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

merging the value count entries of corresponding lists for at least one field from each of at least a first collection and a second collection of the two or more collections to generate a combined list of value count entries, and

aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries, at least some of the distinct field value entries identifying a distinct value from at least one of the value count entries and including information quantifying a number of records in which the distinct value appears for each of the two or more collections;

wherein each value count entry in a list of value count entries corresponding to a particular collection further includes location information identifying respective locations of records within the particular collection of records in which the value appears in the selected field.

37. A method for profiling data stored in a data storage system, the method including:

accessing multiple collections of records stored in the data storage system over an interface coupled to the data storage system to store quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in the selected field and a count of the number of records in which the value appears in the selected field; and

processing the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

merging the value count entries of corresponding lists for at least one field from each of at least a first collection and a second collection of the two or more collections to generate a combined list of value count entries,

aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries, at least some of the distinct field value entries identifying a distinct value from at least one of the value count entries and including information quantifying a number of records in which the distinct value appears for each of the two or more collections, and

reading the value count entries of a corresponding list for at least one field from a third collection of the two or more collections to update the list of distinct field value entries so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first, second, and third collections and include information quantifying a number of records in which the distinct value appears for each of the first, second, and third collections.

38. The method of claim 37, wherein merging the value count entries of the corresponding lists includes sorting the value count entries based on the values from the value count entries.

39. The method of claim 38, wherein sorting the value count entries includes sorting by a primary sort order based on the values from the value count entries, and sorting by a secondary sort order based on identifiers of the two or more collections.

40. A computer program, stored on a computer-readable storage medium, for profiling data stored in a data storage system, the computer program including instructions for causing a computing system to:

access multiple collections of records stored in the data storage system over an interface coupled to the data storage system to store quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in the selected field and a count of the number of records in which the value appears in the selected field; and

process the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

merging the value count entries of corresponding lists for at least one field from each of at least a first collection and a second collection of the two or more collections to generate a combined list of value count entries,

aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries, at least some of the distinct field value entries identifying a distinct value from at least one of the value count entries and including information quantifying a number of records in which the distinct value appears for each of the two or more collections, and

reading the value count entries of a corresponding list for at least one field from a third collection of the two or more collections to update the list of distinct field value entries so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first, second, and third collections and include information quantifying a number of records in which the distinct value appears for each of the first, second, and third collections.

41. The computer program of claim 40, wherein merging the value count entries of the corresponding lists includes sorting the value count entries based on the values from the value count entries.

42. The computer program of claim 41, wherein sorting the value count entries includes sorting by a primary sort order based on the values from the value count entries, and sorting by a secondary sort order based on identifiers of the two or more collections.

43. A computing system for profiling data stored in a data storage system, the computing system including:

an interface coupled to the data storage system configured to access multiple collections of records stored in the data storage system to store quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in the selected field and a count of the number of records in which the value appears in the selected field; and

at least one processor configured to process the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

merging the value count entries of corresponding lists for at least one field from each of at least a first collection and a second collection of the two or more collections to generate a combined list of value count entries,

aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries, at least some of the distinct field value entries identifying a distinct value from at least one of the value count entries and including information quantifying a number of records in which the distinct value appears for each of the two or more collections, and

reading the value count entries of a corresponding list for at least one field from a third collection of the two or more collections to update the list of distinct field value entries so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first, second, and third collections and include information quantifying a number of records in which the distinct value appears for each of the first, second, and third collections.

44. The computing system of claim 43, wherein merging the value count entries of the corresponding lists includes sorting the value count entries based on the values from the value count entries.

45. The computing system of claim 44, wherein sorting the value count entries includes sorting by a primary sort order based on the values from the value count entries, and sorting by a secondary sort order based on identifiers of the two or more collections.

46. A computing system for profiling data stored in a data storage system, the computing system including:

means for accessing multiple collections of records stored in the data storage system to store quantitative information for each of the multiple collections of records, the quantitative information for each particular collection including, for at least one selected field of the records in the particular collection, a corresponding list of value count entries, with each value count entry including a value appearing in the selected field and a count of the number of records in which the value appears in the selected field; and

means for processing the quantitative information of two or more of the collections to generate profiling summary information, the processing including;

merging the value count entries of corresponding lists for at least one field from each of at least a first collection and a second collection of the two or more collections to generate a combined list of value count entries,

aggregating value count entries of the combined list of value count entries to generate a list of distinct field value entries, at least some of the distinct field value entries identifying a distinct value from at least one of the value count entries and including information quantifying a number of records in which the distinct value appears for each of the two or more collections, and

reading the value count entries of a corresponding list for at least one field from a third collection of the two or more collections to update the list of distinct field value entries so that at least some of the distinct field value entries identify a distinct value from value count entries of corresponding lists for the first, second, and third collections and include information quantifying a number of records in which the distinct value appears for each of the first, second, and third collections.