METHOD AND SYSTEM FOR CREATING AND MAINTAINING UNIQUE DATA REPOSITORY

• US 20140195494A1
• Filed: 03/27/2013
• Published: 07/10/2014
• Est. Priority Date: 01/08/2013
• Status: Active Grant

• Current Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Original Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Inventors: GURU, Nishant, PRAKASH, Amit, OJHA, Arjun, AIMA, Garima
• Granted Patent: US 9,342,540 B2
• Time in Patent Office: Days
• US Class Current: 707/692
• CPC Class Codes: G06F 16/16 File or folder operations, e.g. details of user interfaces specifically adapted to file systems
• CPC Class Codes: G06F 16/174 Redundancy elimination performed by the file system management of the data involved in backup or backup restore using de-duplication of the data G06F11/14
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/1756 based on delta files
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2456 Join operations

Abstract

In accordance with the disclosure, there is provided a system and method for creating and maintaining unique data repository comprising a matching process based on a set of predefined matching conditions and thereon performing an action type corresponding to the outcome of matching process. The present disclosure provides for real time data de-duplication and updation of unique data repository to obtain a unified view of unique and matching records.

Claims

1. A computer implemented method of managing and updating a data repository in real time, comprising:

retrieving at least one record from a stream of incoming records maintained in a load repository for matching the retrieved record with a previous record history maintained in a detail repository based on a first uniquely identifying matching key;

performing upon the retrieved record a sequence of matching runs executable on a parallel processing engine, said matching runs further comprising;

a) matching the incoming record against a set of records maintained along with their second associated unique identifier in a master repository, said matching based upon a set of predefined matching conditions and performing at least one action type predefined for each possible matching result;

b) for records found non matching in step a, iteratively performing a matching process between the non matching records and image records, said matching based upon a set of predefined matching conditions and performing at least one action type predefined for each possible matching result;

c) for records found non matching in steps a and b, iteratively performing a self join matching process between the records of the load repository, said matching based upon a set of predefined matching conditions and performing at least one action type predefined for each possible matching result; and

d) for the records found non matching in steps a, b and c, identifying remaining similar matched records based on a criterion.

2. The method of claim 1, wherein the detail repository is configured to maintain the history of all the incoming data record along with the first uniquely identifying matching key and the second unique identifier.

3. The method of claim 1, wherein the matching conditions include at least one of:

a detail match that requires the first identifying matching keys of the incoming record and those of the detail repository to match;

a master match that requires the incoming records and those of the master repository to match on the basis of a batch criteria; and

a same match that requires each of the detail match and the master match condition to satisfy.

4. The method of claim 3, wherein the action types include at least one of:

for the records satisfying the detail, the master and the same match conditions, performing an update action which updates records of the master and the detail repository with content of the incoming data set;

for the records satisfying the detail and the master conditions, performing an auto merge action which updates the records of the detail and the master repository with new content while marking previously linked master record as inactive if no other corresponding match exists for it in the detail repository;

for the records satisfying only the master match, performing an auto consolidate action which adds the incoming record as a new record in the detail repository while updating contents of the incoming record in the master repository;

for the records satisfying only the detail match, performing an update new action type characterized which adds the record in the master repository with a new value of the second unique identifier and thereupon updating for records matching the first uniquely identifiable matching key; and

for the records not satisfying either of the detail match and the master match, performing a new action type which adds the record as new record in the master repository and thereupon in the detail repository with new values of the second unique identifier and the first uniquely identifiable matching key respectively.

5. The method of claim 4, wherein the action types determined for step a includes a combination of the Update, the auto consolidate and the auto merge action type.

6. The method of claim 4, wherein the action types determined for step b includes a combination of the Update, the auto consolidate and the auto merge action type.

7. The method of claim 4, wherein the action types determined for step c includes a combination of the auto consolidate, the auto merge, the update new and the new action type.

8. The method of claim 1, wherein the step d further comprising:

for one or more unmatched records, inserting the corresponding image records, into the master record as a set of new records; and

for the matched records remaining after the steps a, b and c, applying a grouping technique to join all matching records as a group and assigning to each record in the group, a new group identification key.

9. A system for managing and updating a unique data repository in real time, the system comprising:

a load repository configured to store a stream of records, each of the incoming record being associated with a first identifying matching key;

a parallel processing system for retrieving at least one record from the load repository and performing thereupon a sequence of matching runs and coordinating with a master repository that is configured to consolidate and store a set of matching records obtained from the matching run along with a corresponding second unique identifier, wherein said matching runs further include;

in a load-master run, matching the incoming record against a set of records maintained in the master repository, said matching based upon a set of predefined matching conditions and performing at least one action type predefined for each possible matching result;

in a load-image run, for records found non matching in the load master run, iteratively performing a matching process between the non matching records and image records, said matching based upon a set of predefined matching conditions and performing at least one action type predefined for each possible matching result;

in a load-load run, for records found non matching in the load-master and load-image run, iteratively performing a self join matching process between the records of the load repository, said matching based upon a set of predefined matching conditions and performing at least one suited action type predefined for each possible matching result; and

in an image-image run, for the records found non matching in the load-master run, load-image run, and load-load run, identifying for remaining similar matched records based on a criterion.

10. The system of claim 9, wherein the matching conditions include at least one of:

a detail match which requires the first identifying matching keys of the incoming record and those of the detail repository to match;

a master match which requires the incoming records and those of the master repository to match on the basis of a batch criteria; and

a same match which requires each of the detail match and the master match condition to satisfy.

11. The system of claim 9, wherein the action type includes at least one of:

for the records satisfying the detail, the master and the same match conditions, performing an update action which updates records of the master and the detail repository with content of the incoming data set;

for the records satisfying the detail and the master conditions, performing an auto merge action which updates the records of the master and the detail repository with new content while marking previously linked master record as inactive if no other corresponding match exists for it in the detail repository;

for the records satisfying only the master match, performing an auto consolidate action which adds the incoming record as a new record in the detail repository while updating contents of the incoming record in the master repository;

for the records satisfying only the detail match, performing an update new action type which adds the record in the master repository with a new value of the second unique identifier and thereupon updating for records matching the first uniquely identifiable matching key; and

for the records not satisfying either of the detail match and the master match, performing a new action type which adds the record as new record in the master repository and thereupon in the detail repository with new values of the second unique identifier and the first uniquely identifiable matching key respectively.