Intelligent data quality

• US 10,558,629 B2
• Filed: 05/28/2019
• Issued: 02/11/2020
• Est. Priority Date: 05/29/2018
• Status: Active Grant

• Current Assignee: Accenture Global Solutions Limited (Accenture PLC)
• Original Assignee: Accenture Global Services Limited (Accenture PLC)
• Inventors: Yamashita, Sabrina, Martines Neto, Armando, Likhar, Vivek, Da Luz, Acyr
• Primary Examiner(s): Brooks, David T.
• Application Number:
  US16/424,001
• Publication Number:
  US 20190370233A1
• Time in Patent Office: 259 Days
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/903 Querying for retrieval from the web G06F16/953
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Examples of an intelligent data quality application are defined. In an example, the system receives a data quality requirement from a user. The system obtains target data from a plurality of data sources. The system implements an artificial intelligence component sort the target data into a data cascade. The data cascade may include a plurality of attributes associated with the data quality requirement. The system may evaluate the data cascade to identify a data pattern model for each of the attributes. The system may implement a first cognitive learning operation to determine a mapping context from the data cascade and a conversion rule from the data pattern model. The system may establish a data harmonization model corresponding to the data quality requirement by performing a second cognitive learning operation. The system may generate a data cleansing result corresponding to the data quality requirement.

Claims

1. A system comprising:

a processor;

a data profiler coupled to the processor, the data profiler to;

receive a query from a user, the query to indicate a data quality requirement relevant for data management operations;

obtain target data from a plurality of data sources associated with the data quality requirement; and

implement an artificial intelligence component to;

sort the target data into a data cascade, the data cascade to include a plurality of attributes identified by the artificial intelligence component for the target data, each of the attributes from the plurality of attributes being associated with the data quality requirement, wherein the data cascade includes information about an attribute from the plurality of attributes that is linked to another attribute from the plurality of attributes in a sequential manner; and

identify a combination of attributes from the plurality of attributes for generating a data pattern model, the combination including at least one attribute usable for generating the data pattern model;

a data mapper coupled to the processor, the data mapper to;

implement a first cognitive learning operation to;

determine at least one mapping context associated with the data quality requirement from the data cascade and the data pattern model, the mapping context to include a pattern value from the data pattern model and at least one attribute from the data cascade; and

determine a conversion rule from the data pattern model for each of the mapping context associated with the data quality requirement; and

a data cleanser coupled to the processor, the data cleanser to;

obtain the data pattern model for each attribute associated with the data quality requirement;

obtain the conversion rule determined for each of the mapping context associated with the data quality requirement;

establish a data harmonization model corresponding to the data quality requirement by performing a second cognitive learning operation on the obtained data pattern model domain and the obtained conversion rule;

determine a data harmonization index indicative of a level of harmonization achieved in the target data, wherein the data harmonization index provides a quantitative measure of the quality of target data achieved through the selection of the data pattern model, the conversion rule, and the data harmonization model;

modify at least one of the data pattern model, the conversion rule, and the data harmonization model based on the data harmonization index; and

generate a data cleansing result corresponding to the data quality requirement, the data cleansing result comprising the data harmonization model relevant for resolution to the query.

2. The system as claimed in claim 1, wherein the data cleansing result is generated as an electronic document in response to the query of the user.

3. The system as claimed in claim 1, wherein the system modifies any of the data pattern model, the conversion rule, and the data harmonization model for obtaining a consent from the user.

4. The system as claimed in claim 1, wherein the data harmonization model is to further include a harmonized dataset comprising at least a part of the target data from the plurality of sources collated through the implementation of the artificial intelligence component, the first cognitive learning operation, and the second cognitive operation.

5. The system as claimed in claim 1, wherein the data harmonization index indicates a level of harmonization achieved in the target data through the implementation of the artificial intelligence component, the first cognitive learning operation, and the second cognitive operation.

6. The system as claimed in claim 1, wherein the system is to further establish a profiling library, by associating the plurality of attributes included in the data cascade, the mapping context for each attribute, the conversion rule for each mapping context, and the data harmonization model with the data quality requirement.

7. The system as claimed in claim 6, wherein the system is to further deploy the profiling library for validation of the data harmonization model.

8. A method comprising:

receiving, by a processor, a query from a user, the query being indicative of a data quality requirement for data management operations;

obtaining, by the processor, target data from a plurality of data sources associated with the data quality requirement;

implementing, by the processor, an artificial intelligence component to sort the target data into a data cascade, the data cascade to include a plurality of attributes identified by the artificial intelligence component for the target data, each of the attributes from the plurality of attributes being associated with the data quality requirement, wherein the data cascade includes information about an attribute from the plurality of attributes that is linked to another attribute from the plurality of attributes in a sequential manner;

identifying a combination of attributes from the plurality of attributes for generating a data pattern model, the combination including at least one attribute usable for generating the data pattern model;

implementing, by the processor, a first cognitive learning operation to determine at least one mapping context associated with the data quality requirement from the data cascade and the data pattern model, the mapping context to include a pattern value from the data pattern model and at least one attribute from the data cascade;

determining, by the processor, a conversion rule from the data pattern model for each of the mapping context associated with the data quality requirement;

obtaining, by the processor, the data pattern model for each attribute associated with the data quality requirement;

obtaining, by the processor, the conversion rule determined for each of the mapping context associated with the data quality requirement;

establishing, by the processor, a data harmonization model corresponding to the data quality requirement by performing a second cognitive learning operation on the obtained data pattern model domain and the obtained conversion rule;

determining a data harmonization index indicative of a level of harmonization achieved in the target data, wherein the data harmonization index provides a quantitative measure of the quality of target data achieved through the selection of the data pattern model, the conversion rule, and the data harmonization model;

modifying at least one of the data pattern model, the conversion rule, and the data harmonization model based on the data harmonization index; and

generating, by the processor, a data cleansing result corresponding to the data quality requirement, the data cleansing result comprising the data harmonization model relevant for resolution to the query.

9. The method as claimed in claim 8, wherein the method further comprise generating the data cleansing result, by the processor, as an electronic document in response to the query of the user.

10. The method as claimed in claim 8, wherein the method further comprises modifying any of the data pattern model, the conversion rule, and the data harmonization model for obtaining a consent from the user.

11. The method as claimed in claim 8, wherein the data harmonization model is to further include a harmonized dataset comprising at least a part of the target data from the plurality of sources collated through the implementation of the artificial intelligence component, the first cognitive learning operation, and the second cognitive operation.

12. The method as claimed in claim 8, wherein the data harmonization index indicates a level of harmonization achieved in the target data through the implementation of the artificial intelligence component, the first cognitive learning operation, and the second cognitive operation.

13. The method as claimed in claim 8, wherein the method further comprises establishing a profiling library, by associating the plurality of attributes included in the data cascade, the mapping context for each attribute, the conversion rule for each mapping context, and the data harmonization model with the data quality requirement.

14. The method as claimed in claim 13, wherein the method further comprises deploying the profiling library for validation of the data harmonization model.

15. A non-transitory computer readable medium including machine readable instructions that are executable by a processor to:

receive a query from a user, the query being indicative of a data quality requirement associated with data management operations;

obtain target data from a plurality of data sources associated with the data quality requirement;

implement an artificial intelligence component to sort the target data into a data cascade, the data cascade to include a plurality of attributes identified by the artificial intelligence component for the target data, each of the attributes from the plurality of attributes being associated with the data quality requirement, wherein the data cascade includes information about an attribute from the plurality of attributes that is linked to another attribute from the plurality of attributes in a sequential manner;

identify a combination of attributes from the plurality of attributes for generating a data pattern model, the combination including at least one attribute usable for generating the data pattern model;

implement a first cognitive learning operation to determine at least one mapping context associated with the data quality requirement from the data cascade and the data pattern model, the mapping context to include a pattern value from the data pattern model and at least one attribute from the data cascade;

determine a conversion rule from the data pattern model for each of the mapping context associated with the data quality requirement;

obtain the data pattern model for each attribute associated with the data quality requirement;

obtain the conversion rule determined for each of the mapping context associated with the data quality requirement;

establish a data harmonization model corresponding to the data quality requirement by performing a second cognitive learning operation on the obtained data pattern model domain and the obtained conversion rule;

determine a data harmonization index indicative of a level of harmonization achieved in the target data, wherein the data harmonization index provides a quantitative measure of the quality of target data achieved through the selection of the data pattern model, the conversion rule, and the data harmonization model;

modify at least one of the data pattern model, the conversion rule, and the data harmonization model based on the data harmonization index; and

generate a data cleansing result corresponding to the data quality requirement, the data cleansing result comprising the data harmonization model relevant for resolution to the query.

16. The non-transitory computer-readable medium of claim 15, wherein the processor is to generate the data cleansing result as an electronic document in response to the query of the user.

17. The non-transitory computer-readable medium of claim 15, wherein the processor is to modify any of the data pattern model, the conversion rule, and the data harmonization model for obtaining a consent from the user.

18. The non-transitory computer-readable medium of claim 15, wherein the data harmonization model is to further include a harmonized dataset comprising at least a part of the target data from the plurality of sources collated through the implementation of the artificial intelligence component, the first cognitive learning operation, and the second cognitive operation.

19. The non-transitory computer-readable medium of claim 15, wherein the data harmonization index is indicative of a level of harmonization achieved in the at least a part of the target data through the implementation of the artificial intelligence component, the first cognitive learning operation, and the second cognitive operation.

20. The non-transitory computer-readable medium of claim 15, wherein the processor is to further establish a profiling library, by associating the plurality of attributes included in the data cascade, the mapping context for each attribute, the conversion rule for each mapping context, and the data harmonization model with the data quality requirement.