Techniques for integrating validation results in data profiling and related systems and methods

• US 11,068,540 B2
• Filed: 01/25/2018
• Issued: 07/20/2021
• Est. Priority Date: 01/25/2018
• Status: Active Grant

• Current Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Original Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Inventors: Gould, Joel, Rybicki, Stephen George
• Primary Examiner(s): Featherstone, Mark D
• Assistant Examiner(s): Doraiswamy, Ranjit P
• Application Number:
  US15/880,279
• Publication Number:
  US 20190228108A1
• Time in Patent Office: 1,272 Days
• CPC Class Codes: G06F 16/116 Details of conversion of file system types or formats
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/90344 by using string matching techniques
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04

Abstract

According to some aspects, techniques for configuring a data processing system are provided that increase flexibility and efficiency of generation of a data profile of a dataset. The data processing system may produce a value census and a validation census of the dataset in separate processing steps. The value census may then be enriched with contents of the validation census by processing the validation census in a manner that allows matching of field-value pairs of the dataset between the two censuses.

Claims

1. A computer-implemented method of operating a data processing system to generate a data profile based on:

a dataset having an associated record format defining a plurality of fields;

a value census for the dataset comprising a first plurality of values each having an associated field of the plurality of fields and a plurality of first count values, wherein a first count value indicates a number of times a respective field and value combination occurs in the at least one dataset; and

a validation specification comprising a plurality of validation rules defining criteria for invalidity for one or more fields of the plurality of fields, the method comprising;

generating a validation census based at least in part on the dataset and the validation specification, the validation census comprising a second plurality of values each having an associated field of the plurality of fields, an indication of invalidity, and a second value; and

generating a data profile of the at least one dataset based at least in part on the value census and the validation census, wherein generating the data profile comprises;

matching ones of the second plurality of values of the validation census and their associated fields with ones of the first plurality of values of the value census and their associated fields; and

for ones of the first plurality of values of the value census and their associated fields matching at least one of the second plurality of values of the validation census and their associated fields, recording in the data profile the value associated with the field, at least one indication of invalidity based on the validation census, and at least one of the second values.

2. The method of claim 1, wherein the data profile of the dataset specifies, for a first field of the plurality of fields, a number of records of the dataset that have values for the first field that are valid according to the validation specification.

3. The method of claim 1, wherein generating the validation census comprises generating indications of invalidity for one or more records of the dataset.

4. The method of claim 1, wherein the second value indicates a number of records of the dataset that are invalid according to the validation specification.

5. The method of claim 1, wherein the validation census comprises at least two different indications of invalidity that are each associated with a first value of the second plurality of values.

6. The method of claim 1, wherein the data profile of the dataset specifies, for a first field of the plurality of fields, a number of records of the dataset that have values for the first field that are null and invalid according to the validation specification.

7. The method of claim 1, further comprising generating the value census by:

parsing contents of the dataset into a data structure according to the record format;

collating instances of values associated with each field of the plurality of fields; and

recording counts of said instances.

8. A computer system comprising:

at least one processor;

at least one user interface device; and

at least one computer readable medium comprising processor-executable instructions that, when executed, cause the at least one processor to perform a method of generating a data profile based on;

a dataset having an associated record format defining a plurality of fields;

a value census for the dataset comprising a first plurality of values each having an associated field of the plurality of fields and a plurality of first count values, wherein a first count value indicates a number of times a respective field and value combination occurs in the at least one dataset; and

a validation specification comprising a plurality of validation rules defining criteria for invalidity for one or more fields of the plurality of fields, the method comprising;

generating, based at least in part on the dataset and the validation specification, a validation census comprising a second plurality of values each having an associated field of the plurality of fields, an indication of invalidity, and a second value; and

generating, based at least in part on the value census and the validation census, a data profile of the at least one dataset by;

matching ones of the second plurality of values of the validation census and their associated fields with ones of the first plurality of values of the value census and their associated fields; and

for ones of the first plurality of values of the value census and their associated fields matching at least one of the second plurality of values of the validation census and their associated fields, recording in the data profile the value associated with the field, at least one indication of invalidity based on the validation census, and at least one of the second values.

9. The computer system of claim 8, wherein the data profile of the dataset specifies, for a first field of the plurality of fields, a number of records of the dataset that have values for the first field that are valid according to the validation specification.

10. The computer system of claim 8, wherein generating the validation census comprises generating indications of invalidity for one or more records of the dataset.

11. The computer system of claim 8, wherein the second value indicates a number of records of the dataset that are invalid according to the validation specification.

12. The computer system of claim 8, wherein the validation census comprises at least two different indications of invalidity that are each associated with a first value of the second plurality of values.

13. The computer system of claim 8, wherein the data profile of the dataset specifies, for a first field of the plurality of fields, a number of records of the dataset that have values for the first field that are null and invalid according to the validation specification.

14. The computer system of claim 8, wherein the processor-executable instructions further cause the at least one processor to generate the value census by:

parsing contents of the dataset into a data structure according to the record format;

collating instances of values associated with each field of the plurality of fields; and

recording counts of said instances.

15. A computer system for generating a data profile based on:

a dataset having an associated record format defining a plurality of fields;

a value census for the dataset comprising a first plurality of values each having an associated field of the plurality of fields and a plurality of first count values, wherein a first count value indicates a number of times a respective field and value combination occurs in the at least one dataset; and

a validation specification comprising a plurality of validation rules defining criteria for invalidity for one or more fields of the plurality of fields;

comprising;

at least one processor;

means for generating, based at least in part on the dataset and the validation specification, a validation census comprising a second plurality of values each having an associated field of the plurality of fields, and a plurality of indications an indication of invalidity, and a second value; and

means for generating, based at least in part on the value census and the validation census, a data profile of the at least one dataset by;

matching ones of the second plurality of values of the validation census and their associated fields with ones of the first plurality of values of the value census and their associated fields; and

for ones of the first plurality of values of the value census and their associated fields matching at least one of the second plurality of values of the validation census and their associated fields, recording in the data profile the value associated with the field, at least one indication of invalidity based on the validation census, and at least one of the second values.