DATA QUALITY ISSUE DETECTION THROUGH ONTOLOGICAL INFERENCING

• US 20130091084A1
• Filed: 10/11/2011
• Published: 04/11/2013
• Est. Priority Date: 10/11/2011
• Status: Active Grant

• Current Assignee: Leidos Innovations Technology, Inc.
• Original Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Inventors: Lee, Jason E.
• Granted Patent: US 10,460,238 B2
• Time in Patent Office: Days
• US Class Current: 706/46
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

Systems and methods (e.g., utilities) for use in providing automated data quality detection that can be used multiple times across various domains and across multiple data quality spheres. A data structure or schema of an incoming data set is initially mapped to a desired data or knowledge state in a domain ontology made up of a number of TBox statements. Data quality issues in the incoming data set can then be detected as instances of the incoming data set are attempted to be inferenced against or otherwise matched to anticipated TBox statements of the domain ontology.

Claims

1. A method for use in detecting data quality issues in one or more instances of an incoming set of data, the method comprising:

determining a scope of an incoming data set to be receivable at a processing engine;

obtaining a domain ontology that includes a plurality of TBox statements representing a desired state of the incoming data set;

mapping, using a processing engine, the incoming data set to the domain ontology; and

for each instance of the incoming data set;

determining, using the processing engine, whether the instance can be inferenced into an anticipated TBox statement collection of the domain ontology; and

assessing, using the processing engine, whether the instance has at least one data quality issue based on an outcome of the determining.

2. The method of claim 1, wherein the instance can be inferenced into the anticipated TBox statement collection when the instance comprises an ABox statement that is compliant with the anticipated TBox statement collection.

3. The method of claim 1, further comprising:

responsive to the instance being inferenced into the anticipated TBox statement collection;

ascertaining that the instance is free of at least one data quality issue; and

repeating the determining and assessing for another instance of the incoming data set, otherwise;

concluding that the instance has at least one data quality issue;

determining whether the instance can be inferenced into any other TBox statement collections of the domain ontology; and

assessing a type of the at least one data quality issue based on an outcome of the determining.

4. The method of claim 3, further comprising:

ascertaining that the type of the at least one data quality issue comprises a structural and/or formatting issue associated with the instance; and

repeating the determining and assessing for another instance of the incoming data set.

5. The method of claim 4, further comprising:

incrementing, using the processing engine, a statistical count that represents a quantity of structural and/or format issues in the incoming data set responsive to ascertaining that the type of the at least one data quality issue comprises a structural and/or formatting issue associated with the instance.

6. The method of claim 3, further comprising:

ascertaining that the type of the at least one data quality issue comprises a labeling issue associated with the instance; and

repeating the determining and assessing for another instance of the incoming data set.

7. The method of claim 6, further comprising:

incrementing, using the processing engine, a statistical count that represents a quantity of labeling issues in the incoming data set responsive to ascertaining that the type of the at least one data quality issue comprises a labeling issue associated with the instance.

8. The method of claim 1, further comprising:

incrementing, using the processing engine, a statistical count that represents a total quantity of instances in the incoming data set.

9. The method of claim 1, wherein the mapping comprises:

associating a schema of the incoming data set with a schema of the domain ontology.

10. The method of claim 9, wherein the associating comprises:

linking a first structure of the incoming data set schema to the anticipated TBox statement collection of the domain ontology.

11. The method of claim 10, wherein the linking comprises:

binding at least one of a particular character or field of each data segment of an incoming data set to a particular property of the anticipated TBox statement collection.

12. The method of claim 11, wherein each segment comprises a row of data.

13. The method of claim 11, wherein the determining comprises:

comparing the particular character or field of the instance to the particular bound property of the anticipated TBox statement collection.

14. A system for use in detecting data quality issues in one or more instances of an incoming set of data, the system comprising:

a processing module; and

a memory module logically connected to the processing module and comprising a set of computer readable instructions executable by the processing module to;

determine a scope of an incoming data set to be receivable at a processing engine;

obtain a domain ontology that includes a plurality of TBox statements representing a desired state of the incoming data;

map the incoming data set to the domain ontology; and

for each instance of the incoming data set;

determine whether the instance can be inferred into an anticipated TBox statement collection of the domain ontology; and

assess whether the instance has at least one data quality issue based on an outcome of the determining.

15. The system of claim 14, wherein the instance can be inferenced into the anticipated TBox statement collection when the instance comprises an ABox statement that is compliant with the anticipated TBox statement collection.

16. The system of claim 14, wherein responsive to the instance being inferenced into the anticipated TBox statement collection, the computer readable instructions are further executable by the processing module to:

ascertain that the instance is free of at least one data quality issue; and

repeat the set of computer readable instructions that are operable to determine and assess for another instance of the incoming data set;

otherwise, the computer readable instructions are further executable by the processing module to;

conclude that the instance has at least one data quality issue;

determine whether the instance can be inferenced into any other TBox statement collections of the domain ontology; and

assess a type of the at least one data quality issue based on an outcome of the determining.

17. The system of claim 16, wherein the computer readable instructions are further executable by the processing module to:

ascertain that the type of the at least one data quality issue comprises a structural and/or formatting issue associated with the instance; and

repeat the set of computer readable instructions that are operable to determine and assess for another instance of the incoming data set.

18. The system of claim 17, wherein the computer readable instructions are further executable by the processing module to:

increment a statistical count that represents a quantity of structural and/or format issues in the incoming data set responsive to ascertaining that the type of the at least one data quality issue comprises a structural and/or formatting issue associated with the instance.

19. The system of claim 16, wherein the computer readable instructions are further executable by the processing module to:

ascertain that the type of the at least one data quality issue comprises a labeling issue associated with the instance; and

repeat the set of computer readable instructions that are operable to determine and assess for another instance of the incoming data set.

20. The system of claim 19, wherein the computer readable instructions are further executable by the processing module to:

increment, using the processing engine, a statistical count that represents a quantity of labeling issues in the incoming data set responsive to ascertaining that the type of the at least one data quality issue comprises a labeling issue associated with the instance.

21. The system of claim 14, wherein the computer readable instructions are further executable by the processing module to:

increment, using the processing engine, a statistical count that represents a total quantity of instances in the incoming data set.

22. The system of claim 14, wherein the computer readable instructions executable by the processing module to map comprise instructions executable by the processing module to:

associate a schema of the incoming data set with a schema of the domain ontology.

23. The system of claim 22, wherein the computer readable instructions executable by the processing module to associate comprise instructions executable by the processing module to:

link a first structure of the incoming data set schema to the anticipated TBox statement collection of the domain ontology.

24. The system of claim 23, wherein the computer readable instructions executable by the processing module to link comprise instructions executable by the processing module to:

bind at least one of a particular character or field of each data segment of an incoming data set to a particular property of the anticipated TBox statement collection.

25. The system of claim 24, wherein each segment comprises a row of data.

26. The system of claim 24, wherein the computer readable instructions executable by the processing module to determine comprise instructions executable by the processing module to:

compare the particular character or field of the instance to the particular bound property of the anticipated TBox statement collection.