Generating data pattern information

• US 9,652,513 B2
• Filed: 11/30/2015
• Issued: 05/16/2017
• Est. Priority Date: 01/28/2011
• Status: Active Grant

• Current Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Original Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Inventors: Anderson, Arlen
• Primary Examiner(s): Pham, Hung Q
• Application Number:
  US14/954,434
• Publication Number:
  US 20160078100A1
• Time in Patent Office: 533 Days
• Field of Search: None
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/2468 Fuzzy queries
• CPC Class Codes: H03M 7/30 Compression speech analysis-synthesis for redundancy reduction G10L19/00; for image communication H04N; Expansion; Suppression of unnecessary data, e.g. redundancy reduction

Abstract

A data storage system stores at least one dataset including a plurality of records. A data processing system, coupled to the data storage system, processes the plurality of records to produce codes representing data patterns in the records, the processing including: for each of multiple records in the plurality of records, associating with the record a code encoding one or more elements, wherein each element represents a state or property of a corresponding field or combination of fields as one of a set of element values, and, for at least one element of at least a first code, the number of element values in the set is smaller than the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset.

Claims

1. A method, including:

storing, in a data storage system, at least one dataset including a plurality of records; and

processing, in a data processing system coupled to the data storage system, multiple records of the plurality of records to produce codes representing data patterns in the multiple records, the processing including;

for each of the multiple records in the plurality of records, associating with the record a code encoding a plurality of elements, wherein each element represents a state or property of a corresponding field or combination of fields as one of a set of element values, and, for at least one element of at least a first code, the number of element values in the set is smaller than the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset;

determining one or more data patterns characterizing the multiple records based at least in part on the multiple codes associated with the multiple records; and

processing the multiple records in the plurality of records based on the determined one or more data patterns, including at least one of;

processing one or more subsets of the multiple records based on at least one corresponding determined data pattern from the determined one or more data patterns, or determining at least one correlation between states or properties of different fields based on the determined one or more data patterns characterizing the multiple records.

2. The method of claim 1, wherein, for at least one element of the first code, at least two element values in the set represent a state of a field as populated or not populated, respectively.

3. The method of claim 2, wherein the state of not populated includes one of a null state, an absent state, or an empty state.

4. The method of claim 1, wherein, for at least one element of the first code, at least two element values in the set represent a data value occurring in a field being valid or not valid according to a validation rule associated with the field.

5. The method of claim 4, wherein the validation rule defines valid values at least in part by reference to a list of enumerated values.

6. The method of claim 1, wherein, for at least one element of the first code, one or more element values in the set represent a degree to which a data value occurring in a first field matches a target data value.

7. The method of claim 6, wherein the target data value is a data value occurring in a second field.

8. The method of claim 6, wherein at least three element values in the set represent the degree to which the data value occurring in the first field matches the target data value as a mismatch, an exact match, or a fuzzy match, respectively.

9. The method of claim 1, wherein, for at least one element of the first code, at least one element value in the set encodes a data value computed from one or more values occurring in a field or combination of fields of a first record and one or more values occurring in a field or combination of fields of a set of one or more records including at least a second record different from the first record.

10. The method of claim 9, wherein the first record is in a first dataset and the second record is in a second dataset different from the first dataset.

11. The method of claim 1, wherein, for at least one element of at least a second code, at least one element value in the set represents a predetermined data value occurring in a field.

12. The method of claim 11, wherein the number of element values in the set of element values for the element of the second code is equal to the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset.

13. The method of claim 1, wherein processing the one or more subsets of the multiple records includes, for at least one determined data pattern, identifying one or more records representative of that determined data pattern.

14. The method of claim 1, wherein the at least one correlation includes a state of a first field as populated or not populated being correlated to a state of a second field as populated or not populated.

15. A non-transitory computer-readable medium storing a computer program, the computer program including instructions for causing a computer system to:

store, in a data storage system, at least one dataset including a plurality of records; and

process, in at least one processor of the computer system coupled to the data storage system, multiple records of the plurality of records to produce codes representing data patterns in the multiple records, the processing including;

for each of the multiple records in the plurality of records, associating with the record a code encoding a plurality of elements, wherein each element represents a state or property of a corresponding field or combination of fields as one of a set of element values, and, for at least one element of at least a first code, the number of element values in the set is smaller than the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset;

determining one or more data patterns characterizing the multiple records based at least in part on the multiple codes associated with the multiple records; and

processing the multiple records in the plurality of records based on the determined one or more data patterns, including at least one of;

processing one or more subsets of the multiple records based on at least one corresponding determined data pattern from the determined one or more data patterns, or determining at least one correlation between states or properties of different fields based on the determined one or more data patterns characterizing the multiple records.

16. The non-transitory computer-readable medium of claim 15, wherein, for at least one element of the first code, at least two element values in the set represent a state of a field as populated or not populated, respectively.

17. The non-transitory computer-readable medium of claim 16, wherein the state of not populated includes one of a null state, an absent state, or an empty state.

18. The non-transitory computer-readable medium of claim 15, wherein, for at least one element of the first code, at least two element values in the set represent a data value occurring in a field being valid or not valid according to a validation rule associated with the field.

19. The non-transitory computer-readable medium of claim 18, wherein the validation rule defines valid values at least in part by reference to a list of enumerated values.

20. The non-transitory computer-readable medium of claim 15, wherein, for at least one element of the first code, one or more element values in the set represent a degree to which a data value occurring in a first field matches a target data value.

21. The non-transitory computer-readable medium of claim 20, wherein the target data value is a data value occurring in a second field.

22. The non-transitory computer-readable medium of claim 20, wherein at least three element values in the set represent the degree to which the data value occurring in the first field matches the target data value as a mismatch, an exact match, or a fuzzy match, respectively.

23. The non-transitory computer-readable medium of claim 15, wherein, for at least one element of the first code, at least one element value in the set encodes a data value computed from one or more values occurring in a field or combination of fields of a first record and one or more values occurring in a field or combination of fields of a set of one or more records including at least a second record different from the first record.

24. The non-transitory computer-readable medium of claim 23, wherein the first record is in a first dataset and the second record is in a second dataset different from the first dataset.

25. The non-transitory computer-readable medium of claim 15, wherein, for at least one element of at least a second code, at least one element value in the set represents a predetermined data value occurring in a field.

26. The non-transitory computer-readable medium of claim 25, wherein the number of element values in the set of element values for the element of the second code is equal to the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset.

27. The non-transitory computer-readable medium of claim 15, wherein processing the one or more subsets of the multiple records includes, for at least one determined data pattern, identifying one or more records representative of that determined data pattern.

28. The non-transitory computer-readable medium of claim 15, wherein the at least one correlation includes a state of a first field as populated or not populated being correlated to a state of a second field as populated or not populated.

29. A system, the system including:

a data storage system configured to store at least one dataset including a plurality of records; and

a data processing system, coupled to the data storage system, configured to process multiple records of the plurality of records to produce codes representing data patterns in the multiple records, the processing including;

for each of the multiple records in the plurality of records, associating with the record a code encoding a plurality of elements, wherein each element represents a state or property of a corresponding field or combination of fields as one of a set of element values, and, for at least one element of at least a first code, the number of element values in the set is smaller than the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset;

determining one or more data patterns characterizing the multiple records based at least in part on the multiple codes associated with the multiple records; and

processing the multiple records in the plurality of records based on the determined one or more data patterns, including at least one of;

processing one or more subsets of the multiple records based on at least one corresponding determined data pattern from the determined one or more data patterns, or determining at least one correlation between states or properties of different fields based on the determined one or more data patterns characterizing the multiple records.

30. The system of claim 29, wherein, for at least one element of the first code, at least two element values in the set represent a state of a field as populated or not populated, respectively.

31. The system of claim 30, wherein the state of not populated includes one of a null state, an absent state, or an empty state.

32. The system of claim 29, wherein, for at least one element of the first code, at least two element values in the set represent a data value occurring in a field being valid or not valid according to a validation rule associated with the field.

33. The system of claim 32, wherein the validation rule defines valid values at least in part by reference to a list of enumerated values.

34. The system of claim 29, wherein, for at least one element of the first code, one or more element values in the set represent a degree to which a data value occurring in a first field matches a target data value.

35. The system of claim 34, wherein the target data value is a data value occurring in a second field.

36. The system of claim 35, wherein at least three element values in the set represent the degree to which the data value occurring in the first field matches the target data value as a mismatch, an exact match, or a fuzzy match, respectively.

37. The system of claim 29, wherein, for at least one element of the first code, at least one element value in the set encodes a data value computed from one or more values occurring in a field or combination of fields of a first record and one or more values occurring in a field or combination of fields of a set of one or more records including at least a second record different from the first record.

38. The system of claim 37, wherein the first record is in a first dataset and the second record is in a second dataset different from the first dataset.

39. The system of claim 29, wherein, for at least one element of at least a second code, at least one element value in the set represents a predetermined data value occurring in a field.

40. The system of claim 39, wherein the number of element values in the set of element values for the element of the second code is equal to the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset.

41. The system of claim 29, wherein processing the one or more subsets of the multiple records includes, for at least one determined data pattern, identifying one or more records representative of that determined data pattern.

42. The system of claim 29, wherein the at least one correlation includes a state of a first field as populated or not populated being correlated to a state of a second field as populated or not populated.

43. A system, the system including:

means for storing at least one dataset including a plurality of records; and

means for processing multiple records of the plurality of records to produce codes representing data patterns in the multiple records, the processing including;

for each of the multiple records in the plurality of records, associating with the record a code encoding a plurality of elements, wherein each element represents a state or property of a corresponding field or combination of fields as one of a set of element values, and, for at least one element of at least a first code, the number of element values in the set is smaller than the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset;

determining one or more data patterns characterizing the multiple records based at least in part on the multiple codes associated with the multiple records; and

processing the multiple records in the plurality of records based on the determined one or more data patterns, including at least one of;

processing one or more subsets of the multiple records based on at least one corresponding determined data pattern from the determined one or more data patterns, or determining at least one correlation between states or properties of different fields based on the determined one or more data patterns characterizing the multiple records.

44. The system of claim 43, wherein, for at least one element of the first code, at least two element values in the set represent a state of a field as populated or not populated, respectively.

45. The system of claim 44, wherein the state of not populated includes one of a null state, an absent state, or an empty state.

46. The system of claim 43, wherein, for at least one element of the first code, at least two element values in the set represent a data value occurring in a field being valid or not valid according to a validation rule associated with the field.

47. The system of claim 43, wherein, for at least one element of the first code, one or more element values in the set represent a degree to which a data value occurring in a first field matches a target data value.

48. The system of claim 47, wherein the target data value is a data value occurring in a second field.

49. The system of claim 43, wherein, for at least one element of the first code, at least one element value in the set encodes a data value computed from one or more values occurring in a field or combination of fields of a first record and one or more values occurring in a field or combination of fields of a set of one or more records including at least a second record different from the first record.

50. The system of claim 49, wherein the first record is in a first dataset and the second record is in a second dataset different from the first dataset.

51. The system of claim 43, wherein processing the one or more subsets of the multiple records includes, for at least one determined data pattern, identifying one or more records representative of that determined data pattern.

52. The system of claim 43, wherein the at least one correlation includes a state of a first field as populated or not populated being correlated to a state of a second field as populated or not populated.