MANAGING RECORD FORMAT INFORMATION

• US 20110153667A1
• Filed: 11/12/2010
• Published: 06/23/2011
• Est. Priority Date: 11/13/2009
• Status: Active Grant

• Current Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Original Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Inventors: Gould, Joel, Freundlich, Robert, Parmenter, David W., Vigneau, Joyce L., Farver, Jennifer M.
• Granted Patent: US 10,445,309 B2
• Time in Patent Office: Days
• US Class Current: 707/780
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity

Abstract

Data is prepared for processing in a data processing system using format information. Data is received that includes records that have values for fields over an input device or port. A target record format for processing the data is determined. Multiple records are analyzed according to validation tests to determine whether the data matches candidate record formats. Each candidate record format specifies a format for each field, and each validation test corresponds to at least one candidate record format. In response to receiving results of the validation tests, the target record format is associated with the data based on at least one of: a candidate record format for which at least a partial match was determined according to at least one validation test, a parsed record format selected according to a data type associated with the data, and a constructed record format generated from an analysis of data characteristics.

Claims

1. A method for preparing data for processing in a data processing system based on format information in a data storage system, the method including:

receiving data that includes records that each have one or more values for respective fields over an input device or port; and

determining a target record format for processing the data in the data processing system, including

analyzing multiple records in the data according to multiple validation tests to determine whether the data matches one or more candidate record formats stored in the data storage system, each candidate record format specifying a format for each field of a group of one or more fields, and each validation test corresponding to at least one candidate record format stored in the data storage system, and

in response to receiving results of the validation tests, associating the target record format with the data based on at least one of;

a selected candidate record format for which at least a partial match was determined according to at least one validation test corresponding to the selected candidate record format, a parsed record format generated by a parser selected according to a known data type associated with the data, and a constructed record format generated from an analysis of characteristics of the data.

2. The method of claim 1, wherein the target record format is associated with the data based on the parsed record format in response to none of the validation tests determining at least a partial match to one or more of the candidate record formats.

3. The method of claim 2, wherein the known data type associated with the data is known based on a file type of the data.

4. The method of claim 3, wherein the file type of the data corresponds to a file extension.

5. The method of claim 1, wherein the target record format is associated with the data based on the constructed record format in response to:

none of the validation tests determining at least a partial match to one or more of the candidate record formats, and not having a known data type associated with the data.

6. The method of claim 5, wherein generating the constructed record format from an analysis of characteristics of the data includes recognizing tags in the data and parsing the data to determine multiple records based on the recognized tags.

7. The method of claim 5, wherein generating the constructed record format from an analysis of characteristics of the data includes recognizing delimiters in the data and parsing the data to determine multiple records based on the recognized delimiters.

8. The method of claim 5, wherein generating the constructed record format from an analysis of characteristics of the data includes recognizing that the data is in a substantially binary form without tags or delimiters indicating values of multiple records and receiving one or more field identifiers from a user interface.

9. The method of claim 1, wherein analyzing the multiple records in the data according to a first validation test of the multiple validation tests corresponding to a first candidate record format includes applying the first candidate record format to the data to determine values for each record in the formats specified by the first candidate record format for each field.

10. The method of claim 9, wherein determining whether the data matches the first candidate record format includes analyzing the determined values for the multiple records according to the first validation test to determine whether a number of valid values is larger than a predetermined threshold.

11. The method of claim 10, wherein analyzing determined values for a first record of the multiple records according to the first validation test includes performing a corresponding field test on each determined value for each field.

12. The method of claim 11, wherein performing a first field test on a determined value for a first field includes matching a number of characters in the determined value to a predetermined number of characters.

13. The method of claim 11, wherein performing a first field test on a determined value for a first field includes matching the determined value to one of multiple predetermined valid values for the first field.

14. The method of claim 11, wherein the number of valid values is based on a number of records for which the determined value for a given field passes the field test corresponding to the given field.

15. A system for preparing data for processing in a data processing system based on format information in a data storage system, the system including:

means for receiving data that includes records that each have one or more values for respective fields over an input device or port; and

means for determining a target record format for processing the data in the data processing system, including

analyzing multiple records in the data according to multiple validation tests to determine whether the data matches one or more candidate record formats stored in the data storage system, each candidate record format specifying a format for each field of a group of one or more fields, and each validation test corresponding to at least one candidate record format stored in the data storage system, and

in response to receiving results of the validation tests, associating the target record format with the data based on at least one of;

a selected candidate record format for which at least a partial match was determined according to at least one validation test corresponding to the selected candidate record format, a parsed record format generated by a parser selected according to a known data type associated with the data, and a constructed record format generated from an analysis of characteristics of the data.

16. A computer-readable medium storing a computer program for preparing data for processing in a data processing system based on format information in a data storage system, the computer program including instructions for causing a computer to:

receive data that includes records that each have one or more values for respective fields over an input device or port; and

determine a target record format for processing the data in the data processing system, including

analyzing multiple records in the data according to multiple validation tests to determine whether the data matches one or more candidate record formats stored in the data storage system, each candidate record format specifying a format for each field of a group of one or more fields, and each validation test corresponding to at least one candidate record format stored in the data storage system, and

in response to receiving results of the validation tests, associating the target record format with the data based on at least one of;

a selected candidate record format for which at least a partial match was determined according to at least one validation test corresponding to the selected candidate record format, a parsed record format generated by a parser selected according to a known data type associated with the data, and a constructed record format generated from an analysis of characteristics of the data.