Method of conducting data quality analysis

• US 20050108631A1
• Filed: 09/29/2004
• Published: 05/19/2005
• Est. Priority Date: 09/29/2003
• Status: Abandoned Application

• Current Assignee: Data Inventions
• Original Assignee: Data Inventions
• Inventors: Amorin, Antonio C., Figgins, Gary L.
• Application Number:
  US10/953,728
• Publication Number:
  US 20050108631A1
• Time in Patent Office: Days
• US Class Current: 715/234
• CPC Class Codes: G06Q 10/00 Administration; Management

Abstract

A method for creating a data quality report for a given set of source data. The source data is profiled and then analysis is preferably performed at the relation level, metadata level, and data content level analysis. Any inconsistencies noted during analysis are noted with quality tags preferably comprising a common status and a type describing the category of the identified inconsistency. Reports are then generated that describe and summarize the information contained in the quality tags created during the analysis.

Claims

1. A method of analyzing data quality comprising the steps of:

profiling source data;

performing metadata level analysis and creating quality tags to identify problems with metadata;

performing data content level analysis and creating quality tags to identify problems with data;

generating at least one report describing at least a portion of the identified metadata and data problems.

2. The method of claim 1 wherein the source data comprises at least one of a flat file source and a relational file source.

3. The method of claim 1 wherein each quality tag created comprises a common status.

4. The method of claim 3 wherein each quality tag created further comprises a type describing the category of an identified problem.

5. The method of claim 4 wherein each quality tag created further comprises information indicating at least one of:

who created the tag, the date and time the tag was created, a description of the problem found, and example data.

6. The method of claim 1 wherein performing metadata level analysis comprises performing relation level analysis prior to performing analysis on other aspects of metadata.

7. The method of claim 6 wherein performing relation level analysis comprises identifying inconsistencies with record formatting and encoding.

8. The method of claim 6 wherein performing analysis on other aspects of metadata comprises determining at least one of:

whether the data supports the identified key structure,

whether the data supports the expected referential integrity,

whether the null rule is supported by the data,

whether the data supports the documented data type,

whether there is unsupported length associated with a data type,

whether there are unused attributes, and

whether there are constant attributes.

9. The method of claim 1 wherein performing data content level analysis comprises determining at least one of:

whether there is test or garbage data,

whether there is an unusual mixture of alpha and numeric data,

whether there are multiple date patterns in a date field,

whether there are unusual or inconsistent patterns for an attribute,

whether data contains content with different meanings,

whether there is data that may cause a data exception when used in other programs,

whether there is duplicate data,

whether there is inconsistent use of case in the data,

whether there is data that is out if range for an attribute, and

whether there are invalid lookup values.

10. The method of claim 1 wherein generating at least one report comprises exporting information contained in the quality tags to a repository and executing report generation commands to generate said at least one report based upon the quality tags.

11. The method of claim 10 wherein a report is generated that provides an overview of the file and metadata problems identified across an entire project.

12. The method of claim 10 wherein a report is generated that provides an overview of the data quality problems identified across an entire project.

13. The method of claim 10 wherein a report is generated that provides a summary of the number of data quality, metadata, and file problems found for specific relations.

14. The method of claim 10 wherein a report is generated that provides the detailed information stored in the text of any quality tags created for each attribute.

15. The method of claim 10 wherein a report is generated that provides the detailed information stored in the text of any quality tags created for each attribute with metadata problems.

16. The method of claim 10 wherein a report is generated that provides the detailed information stored in the text of any quality tags created at the relation level identifying format problems.

17. A method for analyzing data quality for a given set of source data, the method comprising:

a. profiling source data;

b. performing relation analysis comprising;

i. creating a catalog;

ii. importing metadata into the catalog from a file characterized by a file structure and a file encoding;

iii. comparing the source data with the file structure and noting inconsistencies with at least one quality tag; and

iv. comparing the source data with the file encoding and noting inconsistencies with at least one quality tag;

c. performing metadata analysis comprising;

i. opening an attribute list for the source data; and

ii. comparing the attribute list to the metadata and noting inconsistencies with at least one quality tag;

d. performing data content analysis comprising;

i. opening an attribute list for the source data;

ii. reviewing source data patterns and noting data pattern inconsistencies with at least one quality tag; and

iii. reviewing the source data values and noting inconsistencies with at least one quality tag;

e. generating reports comprising;

i. exporting the catalog to a repository; and

ii. executing report generation commands.