Data quality issue detection through ontological inferencing

• US 10,460,238 B2
• Filed: 10/11/2011
• Issued: 10/29/2019
• Est. Priority Date: 10/11/2011
• Status: Active Grant

• Current Assignee: Leidos Innovations Technology, Inc.
• Original Assignee: Leidos Innovations Technology, Inc.
• Inventors: Lee, Jason E.
• Primary Examiner(s): Waldron, Scott A.
• Assistant Examiner(s): Zidanic, Michael
• Application Number:
  US13/270,945
• Publication Number:
  US 20130091084A1
• Time in Patent Office: 2,940 Days
• Field of Search: None
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

Systems and methods (e.g., utilities) for use in providing automated data quality detection that can be used multiple times across various domains and across multiple data quality spheres. A data structure or schema of an incoming data set is initially mapped to a desired data or knowledge state in a domain ontology made up of a number of TBox statements. Data quality issues in the incoming data set can then be detected as instances of the incoming data set are attempted to be inferenced against or otherwise matched to anticipated TBox statements of the domain ontology.

Claims

1. A method for use in detecting data quality issues in one or more instances of an incoming set of data, the method comprising:

determining a scope of an incoming data set to be receivable at a processor, wherein said determining a scope of the incoming data set comprises automatically perusing a plurality of instances of data structures comprising the incoming data set, and wherein each instance includes one or more entries;

obtaining, based on the determined scope, a domain ontology that includes a plurality of TBox statement collections that collectively comprise metadata describing desired or acceptable properties of data corresponding to the determined scope, and wherein the metadata in each TBox statement collection describes a) a plurality of data types, b) at least one format that each data type should have, and c) an indication of whether or not each format is required in order for the data to be considered compliant with the TBox statement collection, wherein at least one of the formats of at least one of the data types is indicated as being required in order for the data to be considered compliant with the TBox statement collection;

mapping, using the processor, the incoming data set to the domain ontology, wherein said mapping comprises linking specific data structures in the incoming data set to particular TBox statement collections of the obtained domain ontology; and

for each instance of the incoming data set;

identifying an anticipated TBox statement collection of the plurality of TBox statement collections of the domain ontology;

ascertaining, using the processor, whether the instance can be inferenced into the anticipated TBox statement collection of the domain ontology, wherein the instance can be inferenced into the anticipated TBox statement collection when the instance comprises an ABox statement that is compliant with the anticipated TBox statement collection, wherein the instance comprises an ABox statement that is compliant with the anticipated TBox statement collection when at least one data structure in the instance is the at least one of the data types having the at least one of the required formats;

determining, by the processor, that the instance is free of at least one data quality issue responsive to the instance being inferenced into the anticipated TBox statement collection; and

determining, by the processor, that the instance has at least one data quality issue responsive to the instance not being inferenced into the anticipated TBox statement collection, wherein when the processor determines that the instance has at least one data quality issue, the method further includes ascertaining whether the instance can be inferenced into any other TBox statement collections of the domain ontology, and wherein the processor;

determines that the at least one data quality issue comprises a structural and/or formatting issue associated with the instance when it is ascertained that the instance cannot be inferenced into any other TBox statement collections of the domain ontology; and

determines that the at least one data quality issue comprises a labeling issue associated with the instance when it is ascertained that the instance can be inferenced into another TBox statement collection of the domain ontology.

2. The method of claim 1, further comprising:

incrementing, using the processor, a statistical count that represents a quantity of structural, format, and/or labeling issues in the incoming data set responsive to ascertaining that the type of the at least one data quality issue comprises a structural, formatting, and/or labeling issue associated with the instance.

3. The method of claim 1, further comprising:

incrementing, using the processor, a statistical count that represents a total quantity of instances in the incoming data set.

4. The method of claim 1, wherein the linking comprises:

binding at least one of a particular character or field of each data segment of an incoming data set to a particular property of the anticipated TBox statement collection.

5. The method of claim 4, wherein each segment comprises a row of data.

6. The method of claim 4, wherein the determining comprises:

comparing the particular character or field of the instance to the particular bound property of the anticipated TBox statement collection.

7. A system for use in detecting data quality issues in one or more instances of an incoming set of data, the system comprising:

a processor; and

a memory logically connected to the processor and comprising a set of computer readable instructions executable by the processor to;

determine a scope of an incoming data set to be receivable at the processor, wherein the scope of the incoming data set is determined by automatically perusing a plurality of instances comprising the incoming data set, wherein each instance includes one or more entries;

obtain, based on the determined scope, a domain ontology that includes a plurality of TBox statement collections that collectively comprise metadata describing desired or acceptable properties of data elements corresponding to the determined scope, and wherein the metadata in each TBox statement collection describes a) a plurality of data types, b) at least one format that each data type should have, and c) an indication of whether or not each format is required in order for the data to be considered compliant with the TBox statement collection, wherein at least one of the formats of at least one of the data types is indicated as being required in order for the data to be considered compliant with the TBox statement collection;

map the incoming data set to the domain ontology, wherein the incoming data set is mapped to the domain ontology by linking specific data structures in the incoming data set to particular TBox statement collection of the obtained domain ontology; and

for each instance of the incoming data set;

identify an anticipated TBox statement collection of the plurality of TBox statement collections of the domain ontology;

ascertain whether the instance can be inferred into the anticipated TBox statement collection of the domain ontology, wherein the instance can be inferenced into the anticipated TBox statement collection when the instance comprises an ABox statement that is compliant with the anticipated TBox statement collection, wherein the instance comprises an ABox statement that is compliant with the anticipated TBox statement collection when at least one data structure in the instance is the at least one of the data types having the at least one of the required formats;

determine that the instance is free of at least one data quality issue responsive to the instance being inferenced into the anticipated TBox statement collection; and

determine that the instance has at least one data quality issue responsive to the instance not being inferenced into the anticipated TBox statement collection, wherein when it is determined that the instance has at least one data quality issue, the set of computer readable instructions are further executable by the processor to ascertain whether the instance can be inferenced into any other TBox statement collections of the domain ontology, and wherein the set of computer readable instructions are further executable by the processor to;

determine that the at least one data quality issue comprises a structural and/or formatting issue associated with the instance when it is ascertained that the instance cannot be inferenced into any other TBox statement collections of the domain ontology; and

determine that the at least one data quality issue comprises a labeling issue associated with the instance when it is ascertained that the instance can be inferenced into another TBox statement collection of the domain ontology.

8. The system of claim 7, wherein the computer readable instructions are further executable by the processor to:

increment a statistical count that represents a quantity of structural, format, and/or labeling issues in the incoming data set responsive to ascertaining that the type of the at least one data quality issue comprises a structural, formatting, and/or labeling issue associated with the instance.

9. The system of claim 7, wherein the computer readable instructions are further executable by the processor to:

increment a statistical count that represents a total quantity of instances in the incoming data set.

10. The system of claim 7, wherein the computer readable instructions executable by the processor to link comprise instructions executable by the processor to:

bind at least one of a particular character or field of each data segment of an incoming data set to a particular property of the anticipated TBox statement collection.

11. The system of claim 10, wherein each segment comprises a row of data.

12. The system of claim 10, wherein the computer readable instructions executable by the processor to determine comprise instructions executable by the processor to:

compare the particular character or field of the instance to the particular bound property of the anticipated TBox statement collection.

13. A non-transitory computer-readable storage medium storing computer-readable instructions, which when executed by a processor, cause the processor to perform a method that comprises:

determining a scope of an incoming data set to be receivable at the processor, wherein said determining a scope of the incoming data set comprises automatically perusing a plurality of instances of data structures comprising the incoming data set, and wherein each instance includes one or more entries;

obtaining, based on the determined scope, a domain ontology that includes a plurality of TBox statement collections that collectively comprise metadata describing desired or acceptable properties of data corresponding to the determined scope, and wherein the metadata in each TBox statement collection describes a) a plurality of data types, b) at least one format that each data type should have, and c) an indication of whether or not each format is required in order for the data to be considered compliant with the TBox statement collection, wherein at least one of the formats of at least one of the data types is indicated as being required in order for the data to be considered compliant with the TBox statement collection;

mapping the incoming data set to the domain ontology, wherein said mapping comprises linking specific data structures in the incoming data set to particular TBox statement collections of the obtained domain ontology; and

for each instance of the incoming data set;

identifying an anticipated TBox statement collection of the plurality of TBox statement collections of the domain ontology;

ascertaining whether the instance can be inferenced into the anticipated TBox statement collection of the domain ontology, wherein the instance can be inferenced into the anticipated TBox statement collection when the instance comprises an ABox statement that is compliant with the anticipated TBox statement collection, wherein the instance comprises an ABox statement that is compliant with the anticipated TBox statement collection when at least one data structure in the instance is the at least one of the data types having the at least one of the required formats;

determining that the instance is free of at least one data quality issue responsive to the instance being inferenced into the anticipated TBox statement collection; and

determining that the instance has at least one data quality issue responsive to the instance not being inferenced into the anticipated TBox statement collection, wherein when the processor determines that the instance has at least one data quality issue, the method further includes ascertaining whether the instance can be inferenced into any other TBox statement collections of the domain ontology, and wherein the set of computer readable instructions when executed by the processor further cause the processor to;

determine that the at least one data quality issue comprises a structural and/or formatting issue associated with the instance when it is ascertained that the instance cannot be inferenced into any other TBox statement collections of the domain ontology; and

determine that the at least one data quality issue comprises a labeling issue associated with the instance when it is ascertained that the instance can be inferenced into another TBox statement collection of the domain ontology.