CENTRALIZED DATA RECONCILIATION USING ARTIFICIAL INTELLIGENCE MECHANISMS

• US 20190370388A1
• Filed: 05/29/2018
• Published: 12/05/2019
• Est. Priority Date: 05/29/2018
• Status: Active Grant

• Current Assignee: Accenture Global Solutions Limited (Accenture PLC)
• Original Assignee: Accenture Global Solutions Limited (Accenture PLC)
• Inventors: LI, Chung-Sheng, MUNGUIA TAPIA, Emmanuel, FAN, Jingyun, GUNJAN, Abhishek, SHIVARAM, Madhura, BADE, Sawani, VENKATASUBRAMANIYAN, Suresh, SHEKHAR, Saumya
• Granted Patent: US 10,642,869 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/235 Update request formulation
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/288 Entity relationship models
• CPC Class Codes: G06F 16/90344 by using string matching techniques
• CPC Class Codes: G06F 40/20 Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 5/025 Extracting rules from data

Abstract

A centralized data reconciliation system processes at least two data streams transmitting data related to one of a plurality of processes and executes a data reconciliation procedure. Unmatched data records identified during the data reconciliation procedure are further categorized into categorized records based on various reason categories and irreconcilable records which could not be categorized into the reason categories. The irreconcilable records are flagged for user input. The user input is recorded to further train the data reconciliation system. The at least two data streams are initially converted into self-describing data streams from which the entities and entity attributes are extracted using the data models received from the data streams. The data records from the first and second self-describing data streams are mapped. The matched pairs and unmatched pairs are selected from the mappings based on respective confidence scores that are estimated in accordance with the rules of data reconciliation.

Claims

1. A centralized data reconciliation system, comprising:

at least one processor;

at least one non-transitory data storage storing thereon an custom dictionary that includes tokens associated with a first self-describing data stream and a second self-describing data stream, the tokens used for the data matching and the custom dictionary being dynamically updateable based on data streams that are received by the data reconciliation system; and

at least one non-transitory computer readable medium storing machine-readable instructions that cause the at least one processor to;

convert at least two data streams originating at a first data system and a second data system into respective at least two self-describing data streams including the first self-describing data stream and the second self-describing data stream, wherein the first self-describing data stream includes respective data records and a first data model and the second self-describing data stream includes respective data records and a second data model;

map the data records in the first self-describing data stream that include entities and entity attributes to entities and entity attributes in the data records of the second self-describing data stream by employing one or more of the custom dictionary and rules of data reconciliation via one or more two-way matchings;

generate respective confidence scores for the mappings wherein the confidence scores indicate a degree of matching between the mapped data records based at least on rules of data reconciliation;

identify one or more of the data records in the second self-describing data stream that match one or more of the data records in the first self-describing data stream from the mappings based at least on the confidence scores;

determine unmatched data records from the data records from the first and the second self-describing data streams based at least on the confidence scores;

classify the unmatched data records into categorized records and irreconcilable records, the categorized records are categorized into one or more reason categories, and the irreconcilable records including the unmatched data records that could not be categorized into the reason categories;

generate one or more of reasons and recommendations for at least a subset of the categorized records; and

automatically update one or more of the custom dictionary, the reason categories and the rules of data reconciliation based on user inputs received for the irreconcilable records for which the reasons and recommendations could not be generated.

2. The centralized data reconciliation system of claim 1, the first data model and the second data model include one or more of entity names, data types and compression schemes used.

3. The centralized data reconciliation system of claim 1, wherein the non-transitory computer readable medium further storing machine-readable instructions that cause the at least one processor to:

extract entities and attributes of the entities from the data records of the first self-describing data stream and the second self-describing data stream using natural language processing (NLP) techniques.

4. The centralized data reconciliation system of claim 3, wherein the non-transitory computer readable medium further storing machine-readable instructions that cause the at least one processor to:

map one or more attributes of the entities in the first self-describing data stream to one or more attributes of the entities in the second self-describing data stream, the mapping provided by the custom dictionary.

5. The centralized data reconciliation system of claim 4, wherein the non-transitory computer readable medium further storing machine-readable instructions that cause the at least one processor to:

retrieve the rules of data reconciliation that are employed to match entities from the first self-describing data stream to entities in the second self-describing data stream, the matching of data records based at least on one of the entity attributes; and

generate a respective one of the confidence scores for each of the mappings.

6. The centralized data reconciliation system of claim 1, wherein the at least two self-describing data streams include at least three self-describing data streams including the first self-describing data stream, the second self-describing data stream and an intermediate self-describing data stream and wherein the non-transitory computer readable medium further storing machine-readable instructions that cause the at least one processor to:

decompose a three-way matching process between the first self-describing data stream to the intermediate data stream and to the second self-describing data stream to two two-way matching processes.

7. The centralized data reconciliation system of claim 1, wherein the instructions for identifying matching data records from the mappings further comprise machine-readable instructions that cause the at least one processor to:

compare the confidence scores for categorizing the mappings into the matched records and the unmatched data records, the matched data records having the respective confidence scores above a first confidence threshold and the unmatched data records having the respective confidence scores below the first confidence threshold.

8. The centralized data reconciliation system of claim 1, the non-transitory computer readable medium storing machine-readable instructions that cause the at least one processor to:

record user input regarding user actions pertaining to the irreconcilable records; and

provide the user input for training the data reconciliation system in resolving the irreconcilable records.

9. The centralized data reconciliation system of claim 8, wherein the non-transitory computer readable medium further storing machine-readable instructions that cause the at least one processor to:

flag for human review the irreconcilable records based on a comparison of respective second confidence levels of the unmatched data records with a second confidence threshold.

10. The centralized data reconciliation system of claim 1, wherein the non-transitory computer readable medium further storing machine-readable instructions that cause the at least one processor to:

receive user feedback to results output by the centralized data reconciliation system.

11. The centralized data reconciliation system of claim 10, wherein the non-transitory computer readable medium further storing machine-readable instructions that cause the at least one processor to:

update one or more of the custom dictionary, the rules of data reconciliation and reason categories based at least on the user feedback.

12. The centralized data reconciliation system of claim 1, wherein locations of the respective data records are included within the at least two self-describing data streams as semantic based data source specifications.

13. A method for data reconciliation comprising:

receiving at least two data streams including at least a first data stream and a second data stream from a first data system and a second data system respectively;

building respective feature vectors from the first data stream and the second data stream;

converting the first data stream and second data stream into respective self-describing data streams that each includes a respective data model based on the feature vectors;

extracting entities and attributes of entities to be matched from the self-describing data streams using the respective data models;

mapping the entities from a first one of the self-describing data streams to a second one of the self-describing data streams by employing an custom dictionary that enables mapping the entities in one or more two-way matches using rules of data reconciliation;

estimating a confidence score for each of the mappings;

identifying matched records and unmatched records from the mappings based on a comparison of the confidence scores with a confidence score threshold;

categorizing a subset of the unmatched records into one or more reason categories and another subset of the unmatched records that could not be categorized as irreconcilable records;

framing one or more hypotheses and a respective discrepancy confidence score for each of the hypotheses, the discrepancy confidence score being indicative of a confidence level of the hypotheses for the categorized records based on one or more of the rules of data reconciliation that were not fulfilled by the irreconcilable records; and

generating a report including one or more of a reason and a recommendation for the categorized records having the respective discrepancy confidence scores above a confidence threshold; and

flagging for user intervention, the irreconcilable records having the respective discrepancy confidence scores below a confidence threshold.

14. The method for data reconciliation of claim 13 further including:

recording user input received for one or more of the irreconcilable records having the respective confidence scores below the confidence threshold and the matched records.

15. The method of claim 13, wherein converting the at least two data streams into respective self-describing data streams further comprises:

generating for the self-describing data streams respective data models including the entities and the attributes of the entities; and

including with the self-describing data streams, the respective data models.

16. The method of claim 13, wherein extracting entities and attributes of entities further comprises:

employing one or more of text matching and natural language processing (NLP) techniques for mapping the entities from the first data stream to the entities in the second data stream.

17. The method of claim 13, wherein the at least two data streams include four data streams and matching the entities from the first self-describing data stream to the second self-describing data stream further comprises:

decomposing a match between the first self-describing data stream and the second self-describing data stream into three two-way matches that include at least a mapping from the first self-describing data stream to a first self-describing intermediate data stream and another mapping from a second self-describing intermediate data stream to the second self-describing intermediate data stream.

18. A non-transitory computer-readable storage medium comprising machine-readable instructions that cause a processor to:

convert at least two data streams including a first data stream and a second data stream originating respectively at a first data system and a second data system into respective self-describing data streams including a first self-describing data stream and a second self-describing data stream, wherein the first self-describing data stream includes respective data records and a first data model and the second self-describing data stream includes respective data records and a second data model;

map the data records in the first self-describing data stream that include entities and entity attributes to entities and entity attributes in the data records of the second self-describing data stream by employing one or more of an custom dictionary and rules of data reconciliation via one or more two-way matchings;

generate respective confidence scores for the mappings wherein the confidence scores indicate a degree of matching between the mapped data records based at least on rules of data reconciliation;

identify one or more of the data records in the second self-describing data stream that match one or more of the data records in the first self-describing data stream from the mappings based at least on the confidence scores;

determine unmatched data records from the data records from the first and the second self-describing data streams based at least on the confidence scores;

classify the unmatched records into categorized records and irreconcilable records, the categorized records being categorized into one or more reason categories, and the irreconcilable records include the unmatched data records that could not be categorized into the reason categories;

generate one or more of reasons and recommendations for at least a subset of the irreconcilable records; and

automatically update one or more of the custom dictionary, the reason categories and the rules of data reconciliation based on user inputs received for the irreconcilable records for which the reasons and recommendations could not be generated.

19. A non-transitory computer-readable storage medium of claim 18, wherein the instructions for converting the at least two data streams further comprising machine-readable instructions that cause a processor to:

build respective feature vectors from the first data stream and the second data stream; and

convert the first and second data streams into respective the self-describing data streams that each includes a respective data model based on the feature vectors.

20. A non-transitory computer-readable storage medium of claim 18, wherein the instructions for mapping the data records further comprising machine-readable instructions that cause a processor to:

employing one or more of text matching and natural language processing (NLP) techniques for mapping the entities from the first data stream to the entities in the second data stream.