GENERATING DATA PATTERN INFORMATION

• US 20120197887A1
• Filed: 01/27/2012
• Published: 08/02/2012
• Est. Priority Date: 01/28/2011
• Status: Active Grant

• Current Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Original Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Inventors: Anderson, Arlen
• Granted Patent: US 9,449,057 B2
• Time in Patent Office: Days
• US Class Current: 707/736
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/2468 Fuzzy queries
• CPC Class Codes: H03M 7/30 Compression speech analysis-synthesis for redundancy reduction G10L19/00; for image communication H04N; Expansion; Suppression of unnecessary data, e.g. redundancy reduction

Abstract

A data storage system stores at least one dataset including a plurality of records. A data processing system, coupled to the data storage system, processes the plurality of records to produce codes representing data patterns in the records, the processing including: for each of multiple records in the plurality of records, associating with the record a code encoding one or more elements, wherein each element represents a state or property of a corresponding field or combination of fields as one of a set of element values, and, for at least one element of at least a first code, the number of element values in the set is smaller than the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset.

Claims

1. A method, including:

storing, in a data storage system, at least one dataset including a plurality of records; and

processing, in a data processing system coupled to the data storage system, the plurality of records to produce codes representing data patterns in the records, the processing including;

for each of multiple records in the plurality of records, associating with the record a code encoding one or more elements, wherein each element represents a state or property of a corresponding field or combination of fields as one of a set of element values, and, for at least one element of at least a first code, the number of element values in the set is smaller than the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset.

2. The method of claim 1, wherein, for at least one element of the first code, at least two element values in the set represent a state of a field as populated or not populated, respectively.

3. The method of claim 2, wherein the state of not populated includes one of a null state, an absent state, or an empty state.

4. The method of claim 3, wherein the null state includes a state of the field having a predetermined value.

5. The method of claim 4, wherein the predetermined value includes one or more space characters.

6. The method of claim 3, wherein the absent state includes the field being missing in a record.

7. The method of claim 3, wherein the empty state includes the field having no corresponding data value.

8. The method of claim 1, wherein, for at least one element of the first code, at least two element values in the set represent a data value occurring in a field being valid or not valid according to a validation rule associated with the field.

9. The method of claim 8, wherein the validation rule defines valid values based at least on part on a data type of the data value.

10. The method of claim 8, wherein the validation rule defines valid values at least in part by reference to a list of enumerated values.

11. The method of claim 1, wherein, for at least one element of the first code, one or more element values in the set represent a degree to which a data value occurring in a first field matches a target data value.

12. The method of claim 11, wherein the target data value is a data value occurring in a second field.

13. The method of claim 11, at least three element values in the set represent the degree to which the data value occurring in the first field matches the target data value as a mismatch, an exact match, or a fuzzy match, respectively.

14. The method of claim 1, wherein, for at least one element of the first code, at least one element value in the set encodes a data value computed from one or more values occurring in a field or combination of fields of a first record and one or more values occurring in a field or combination of fields of a set of one or more records including at least a second record different from the first record.

15. The method of claim 14, wherein the first record is in a first dataset and the second record is in a second dataset different from the first dataset.

16. The method of claim 1, wherein, for at least one element of at least a second code, at least one element value in the set represents a predetermined data value occurring in a field.

17. The method of claim 16, wherein the number of element values in the set of element values for the element of the second code is equal to the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset.

18. A computer-readable storage medium storing a computer program, the computer program including instructions for causing a computer system to:

store, in a data storage system, at least one dataset including a plurality of records; and

process, in at least one processor of the computer system coupled to the data storage system, the plurality of records to produce codes representing data patterns in the records, the processing including;

for each of multiple records in the plurality of records, associating with the record a code encoding one or more elements, wherein each element represents a state or property of a corresponding field or combination of fields as one of a set of element values, and, for at least one element of at least a first code, the number of element values in the set is smaller than the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset.

19. A system, the system including:

a data storage system configured to store at least one dataset including a plurality of records; and

a data processing system, coupled to the data storage system, configured to process the plurality of records to produce codes representing data patterns in the records, the processing including;

for each of multiple records in the plurality of records, associating with the record a code encoding one or more elements, wherein each element represents a state or property of a corresponding field or combination of fields as one of a set of element values, and, for at least one element of at least a first code, the number of element values in the set is smaller than the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset.

20. A system, the system including:

means for storing at least one dataset including a plurality of records; and

means for processing the plurality of records to produce codes representing data patterns in the records, the processing including;

for each of multiple records in the plurality of records, associating with the record a code encoding one or more elements, wherein each element represents a state or property of a corresponding field or combination of fields as one of a set of element values, and, for at least one element of at least a first code, the number of element values in the set is smaller than the total number of data values that occur in the corresponding field or combination of fields over all of the plurality of records in the dataset.