Automated system and method of data scrubbing

• US 9,146,945 B2
• Filed: 03/18/2013
• Issued: 09/29/2015
• Est. Priority Date: 04/03/2012
• Status: Active Grant

• Current Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Original Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Inventors: Kumar, Rajive, Pandey, Vinit, Rastogi, Sandhi, Sharma, Tuhina
• Primary Examiner(s): THAI, HANH B
• Application Number:
  US13/845,936
• Publication Number:
  US 20130262406A1
• Time in Patent Office: 925 Days
• Field of Search: 707687-692
• US Class Current: 1/1
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database

Abstract

A system and method enabling automated data cleansing and scrubbing at the attribute level is disclosed. A consolidated view may be provided of the scrubbed data or narratives that gets promoted to a final copy and the data or narratives received from multiple sources on a single user interface.

Claims

1. A method for performing data scrubbing at attribute level, comprising:

receiving, by a processor, data containing at least one significant attribute or at least one non-significant attribute and associated values from distributed data sources, the data sources being assigned weight against each value of the at least one significant attribute or the at least one non-significant attribute; and

applying, by the processor, a ranking matrix process to the received data, the ranking matrix process comprising;

for sources referring to different values for a significant attribute or a non-significant attribute, computing a combined weight therefrom;

in response to the combined weight of the significant attribute exceeding a predetermined promotion threshold value, determining if the combined weight exceeds a predetermined confirmation threshold value;

in response to the combined weight of the non-significant attribute and the significant attribute exceeding the predetermined promotion threshold value and the predetermined confirmation threshold value respectively, promoting values associated with the non-significant attribute and the significant attribute to a final templated copy;

in response to the combined weight of the significant attribute or the non-significant attribute being less than the predetermined promotion threshold value, computing a total weight of all values for the significant attribute or the non-significant attribute from all sources; and

in response to the total weight exceeding a predetermined task threshold, raising a work item for a user to create a manual source and reapplying the ranking matrix process.

2. The method of claim 1, wherein the at least one significant attribute corresponds to Mandatory or Mandatory to send significance levels and the at least one non-significant attribute corresponds to Optional or View only significance levels.

3. The method of claim 1, further comprising:

performing a preliminary mapping including at least one of source mapping, event mapping and security mapping;

performing duplicate drop check, heldness check; and

performing a narrative matching process before applying the ranking matrix process.

4. The method of claim 1, wherein the ranking matrix process employs a combination of a ranking attribute rule and a source weighting rule.

5. The method of claim 4, wherein the ranking attribute rule comprises a set of rules determining a list of attributes to be present in the final templated copy, the set of rules including:

significance level of each attribute, the promotion threshold value and the confirmation threshold, value, task threshold value event mapping rule, source priority rule, narrative match rule, completeness and confirmation rule, constraint rule, and a source weighting rule assigning the weight to the sources against each value.

6. The method of claim 1, wherein the combined weight is computed by:

selecting the distributed sources referring the different values for the significant attribute or the non-significant attribute to form a group;

determining the groups, so formed, having highest and second highest weights as a first highest group weight and a second highest group weight; and

subtracting the second highest group weight from the first highest group weight to derive the combined weight.

7. The method of claim 1, wherein the total weight is the sum of the weights of the values for the significant attribute or the non-significant attribute from each of the sources.

8. The method of claim 1, wherein the final templated copy along with the significant attribute or the non-significant attribute and the associated values obtained from the distributed sources are displayed on a single screen.

9. The method of claim 1, further comprising computing of a processing status of the final templated copy based on completeness and confirmation status of the significant attribute and the non-significant attribute.

10. The method of claim 1, further comprising displaying on a graphical user interface a summary section to display key attributes of an event and a main section to display source headings, attribute headings, the final templated copy and incoming data, and a toolbar section adapted to perform plurality of icon-based operations responsive to the summary section and the main section.

11. The method of claim 1, wherein the ranking process further comprises:

in response to the combined weight of the significant attribute being less than the predetermined confirmation threshold, verifying status of the significant attribute being changed from confirmed to unconfirmed; and

promoting values of the significant attribute to a golden copy if the status of the significant attribute remains confirmed or reapplying the ranking matrix process if the status of the significant attribute is changed to unconfirmed.

12. A system for performing data scrubbing, comprising:

a hardware processor; and

a memory storing instructions, wherein the hardware processor is configured by the instructions to;

provide an input interface configured to receive, for an event, data containing at least one significant attribute or at least one non-significant attribute and associated values from distributed data sources;

apply a ranking matrix process for determining values associated with the at least one significant attribute or the at least one non-significant attribute to be promoted to a final templated copy based upon a combination of a predefined ranking attribute rule and a source weighting rule, wherein the ranking matrix process comprises;

for sources referring to different values for a significant attribute or a non-significant attribute, computing a combined weight therefrom;

in response to the combined weight of the significant attribute exceeding a predetermined promotion threshold value, determining if the combined weight exceeds a predetermined confirmation threshold value;

in response to the combined weight of the non-significant attribute and the significant attribute exceeding the predetermined promotion threshold value and the predetermined confirmation threshold value respectively, promoting values associated with the non-significant attribute and the significant attribute to a final templated copy;

in response to the combined weight of the significant attribute or the non-significant attribute being less than the predetermined promotion threshold value, computing a total weight of all values for the significant attribute or the non-significant attribute from all sources; and

in response to the total weight exceeding a predetermined task threshold, raising a work item for a user to create a manual source and reapplying the ranking matrix process; and

displaying the final templated copy along with the data received from the distributed sources on a graphical user interface, the graphical interface including a summary section to display key attributes of the event and a main section to display source headings, attribute headings, the final templated copy and incoming data, and a toolbar section adapted to perform a plurality of icon-based operations responsive to the summary section and the main section.

13. The system of claim 12, wherein the summary section displays the key attributes including security number and description, event type, attribute significance, depository, event processing status, default flag, maturity flag, the ranking matrix information, user information and indication of incoming data under processing.

14. The system of claim 12, wherein the main section displays the attribute headings including attributes that get promoted to the final templated copy, associated significance levels, reference information for the final templated copy and the incoming data, the processing status thereof, receipt date, last modified date and message content.

15. The system of claim 12, wherein the main section further displays significance level of each attribute with specific font characteristics.

16. The system of claim 12, wherein the main section displays the source headings including names of the distributed sources and a check box positioned next to a source name displaying a content corresponding to an underlying function of a source selected.

17. The system of claim 12, wherein the final templated copy and the incoming data are rendered with a plurality of icons responsive to show status of an attribute corresponding to any conflict raised, user intervention, pending authorization, promotion of attribute to the final templated copy and absence of data from the source.

18. The system of claim 12, wherein the plurality of icon based operations responsive to the main section corresponds to:

promoting standard data directly to the final templated copy by the user, creating duplicate or making modifications to the event for the source selected in the source heading section, displaying raw messages, audit details, option mismatch screen, highlighting erroneous attributes along with associated error details and underlying functions associated with user resolution of raised conflicts or edits to data directly promoted to the final templated copy.

19. The system of claim 12, wherein the ranking process further comprises:

in response to the combined weight of the significant attribute being less than the predetermined confirmation threshold, verifying status of the significant attribute being changed from confirmed to unconfirmed; and

promoting values of the significant attribute to a golden copy if the status of the significant attribute remains confirmed or reapplying the ranking matrix process if the status of the significant attribute is changed to unconfirmed.

20. A non-transitory computer readable medium embodying a program executable in a computer for performing data scrubbing at attribute level, the program comprising computer executable instructions for:

receiving data containing at least one significant attribute or at least one non-significant attribute and associated values from distributed data sources, the data sources being assigned weight against each value of the at least one significant attribute or the at least one non-significant attribute; and

applying a ranking matrix process to the received data, the ranking matrix process comprising;

for sources referring to different values for a significant attribute or a non-significant attribute, computing a combined weight therefrom;

in response to the combined weight of the significant attribute exceeding a predetermined promotion threshold value, determining if the combined weight exceeds a predetermined confirmation threshold value;

in response to the combined weight of the non-significant attribute and the significant attribute exceeding the predetermined promotion threshold value and the predetermined confirmation threshold value respectively, promoting values associated with the non-significant attribute and the significant attribute to a final templated copy;

in response to the combined weight of the significant attribute or the non-significant attribute being less than the predetermined promotion threshold value, computing a total weight of all values for the significant attribute or the non-significant attribute from all sources; and

in response to the total weight exceeding a predetermined task threshold, raising a work item for a user to create a manual source and reapplying the ranking matrix process.