AUTOMATED SYSTEM AND METHOD OF DATA SCRUBBING

• US 20130262406A1
• Filed: 03/18/2013
• Published: 10/03/2013
• Est. Priority Date: 04/03/2012
• Status: Active Grant

• Current Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Original Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Inventors: KUMAR, Rajive, PANDEY, Vinit, RASTOGI, Sandhi, SHARMA, Tuhina
• Granted Patent: US 9,146,945 B2
• Time in Patent Office: Days
• US Class Current: 707/692
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database

Abstract

A system and method enabling automated data cleansing and scrubbing at the attribute level is disclosed. A consolidated view may be provided of the scrubbed data or narratives that gets promoted to a final copy and the data or narratives received from multiple sources on a single user interface.

Claims

1. A computer implemented method for performing data scrubbing at attribute level, comprising:

receiving data containing at least one significant or non significant attribute and associated values from distributed data sources, the sources being assigned weight against each of the attribute value; and

applying a ranking matrix process to the received data, the ranking process comprising;

for sources referring to different values for the same attribute, computing a combined weight therefrom;

in response to the combined weight of the significant attribute exceeding a predetermined promotion threshold value, determining if said combined weight exceeds a predetermined confirmation threshold value,

in response to the combined weight of the non significant attribute exceeding the promotion threshold value, promoting the non significant attribute to a final templated copy;

in response to the combined weight of the significant or non significant being less than the confirmation threshold value, computing a total weight of all values for the attribute from all sources; and

in response to the total weight exceeding a predetermined task threshold, raising a work item for a user to create a manual source and reapplying the ranking matrix process.

2. The method of claim 1, wherein the significant attribute corresponds to Mandatory or Mandatory to send significance levels and the non significant attribute corresponds to Optional or View only significance levels.

3. The method of claim 1, further comprising performing a preliminary mapping including at least one source mapping, event mapping and security mapping;

performing duplicate drop check, heldness check; and

performing narrative matching process before applying the ranking matrix process.

4. The method of claim 1, wherein the ranking matrix process employs a combination of a ranking attribute rule and a source weighting rule.

5. The method of claim 4, wherein the ranking attribute rule comprises a set of rules determining a list of attributes to be present in the final templated copy, the rules including:

significance level of each attribute, the promotion and confirmation threshold values, task threshold value event mapping rule, source priority rule, narrative match rule, completeness and confirmation rule, constraint rule, and the source weighting rule assigns weight to sources against each of the attribute value.

6. The method of claim 1, wherein the combined weight is computed by:

selecting the distributed sources providing same attribute value to form one group;

determining the groups, so formed, having highest and second highest weights; and

subtracting the second highest group weight from the first highest group weight to derive the combined weight.

7. The method of claim 1, wherein the total weight is sum of weight of the attributes from each of the source.

8. The method of claim 1, wherein the final templated copy along with the attributes and the associated value obtained from the distributed sources is displayed on a single screen.

9. The method of claim 1, further comprising computing of a processing status of the final templated copy based on completeness and conformation status of the significant attributes.

10. The method of claim 1, further comprising displaying on the graphical user interface a summary section to display key attributes of an event and a main section to display source headings, attribute headings, the final templated copy and incoming data, and a toolbar section adapted to perform plurality of icon-based operations responsive to the summary and main section.

11. A system for performing data scrubbing, comprising:

a hardware processor; and

a memory storing instructions, wherein the hardware processor is configured by the instructions to;

provide an input interface configured to receive, for an event, data containing at least one significant or non significant attribute and associated values from distributed data sources;

apply a ranking matrix process for determining attribute values to be promoted to final templated copy based upon a combination of predefined ranking attribute rule and source weighting rule; and

displaying the final templated copy along with the data received from the distributed sources, the graphical interface including a summary section to display key attributes of the event and a main section to display source headings, attribute headings, final templated copy and incoming data, and a toolbar section adapted to perform plurality of icon-based operations responsive to the summary and main section.

12. The system of claim 11, wherein the summary section shows the key attributes including:

security number and description, event type, attribute significance, depository, event processing status, default flag, maturity flag, the ranking matrix information, user information and indication of incoming data under processing.

13. The system of claim 11, wherein the main section displays the attribute headings including all the attributes that get promoted to the final templated copy, associated significance levels, reference information for final templated copy and incoming data, the processing status thereof, receipt date, last modified date and message content.

14. The system of claim 11, wherein the main section further displays significance level of each attribute with specific font characteristics.

15. The system of claim 11, wherein the main section displays the source headings including names of the distributed sources and a check box positioned next to the source name displaying a content corresponding to an underlying function of the selected source.

16. The system of claim 11, wherein the final templated copy and incoming data is rendered with plurality of icons responsive to show status of attribute corresponding to any conflict raised, user intervention, pending authorization, promotion of attribute to the final templated copy and absence of data from the source.

17. The system of claim 11, wherein the plurality of icon based operations responsive to the main section corresponds to:

promoting standard data directly to the final templated copy by the user, creating duplicate or making modifications to the event for the source selected in the source heading section, displaying raw messages, audit details, option mismatch screen, highlighting erroneous attributes along with associated error details and underlying functions associated with user resolution of raised conflicts or edits to data directly promoted to the final templated copy.