Methods and Systems for Assessing Data Quality

• US 20130073594A1
• Filed: 09/19/2011
• Published: 03/21/2013
• Est. Priority Date: 09/19/2011
• Status: Active Grant

• Current Assignee: Citigroup Technology, Inc. (Citigroup Incorporated)
• Original Assignee: Citigroup Technology, Inc. (Citigroup Incorporated)
• Inventors: Jugulum, Rajesh, Gray, Don, Ramachandran, Rajalakshmi, Singh, Jagmeet, Granese, Robert A., Brzozowski, Kenneth, Joyce, Harold Ian
• Granted Patent: US 10,248,672 B2
• Time in Patent Office: Days
• US Class Current: 707/802
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06Q 40/00 Finance; Insurance; Tax strategies; Processing of corporate or income taxes

Abstract

Methods and systems for assessing data involve, collecting samples of data elements from a database storing a population of data elements representing attributes of each numerous different financial transactions. Critical data elements from the collected samples are determined. Data quality rules are built and data dimensions are calculated for the critical data elements. A quality of data within the critical data elements for different data quality dimensions is monitored. Critical data elements that produce a high number of outliers are identified and causes for the outliers are identified. Thereafter, a corrective action plan to address a solution for the causes for the outliers may be developed and executed.

Claims

1. A method of data quality assessment, comprising:

collecting, using a microprocessor, samples of data elements from a database storing a population of data elements representing attributes of each of a plurality of different financial transactions;

determining, using the microprocessor, critical data elements from the collected samples of data elements;

building, using the microprocessor, data quality rules and calculating, using the microprocessor, data dimensions for the critical data elements;

monitoring, using the microprocessor, a quality of data within the critical data elements for different data quality dimensions;

identifying, using the microprocessor, critical data elements that produce a pre-defined high number of outliers;

identifying, using the microprocessor, causes for the outliers; and

developing and executing a corrective action plan to address a solution for the identified causes for the outliers.

2. The method of claim 1, wherein at least some of said attributes of each of said transactions are interrelated in mathematical terms.

3. The method of claim 1, wherein collecting the samples further comprises finding representative samples from said population of data elements using methodologies consisting of at least one of stratified sampling and random sampling.

4. The method of claim 1, wherein determining the critical data elements further comprises rationalizing an order of analysis of proposed critical data elements.

5. The method of claim 4, wherein rationalizing the order of analysis further comprises rationalizing the order of analysis of proposed critical data elements based on a rationalization matrix.

6. The method of claim 5, wherein rationalizing the order of analysis based on the rationalization matrix further comprises rationalizing the order of analysis of proposed critical data elements based on a rationalization matrix for a plurality of proposed critical data elements ranked according to weighted ranking criteria.

7. The method of claim 1, further comprising determining the critical data elements based at least in part on preliminary analytics comprising at least one of correlation and regression analysis.

8. The method of claim 1, further comprising determining the critical data elements based at least in part on executing pattern recognition and attribute dependency algorithms.

9. The method of claim 8, further comprising determining the critical data elements based at least in part on executing the pattern recognition and attribute dependency algorithms executing in a pre-defined sequence.

10. The method of claim 7, wherein determining the critical data elements based at least in part on preliminary analytics comprising correlation analysis further comprises performing the correlation analysis to yield a correlation matrix.

11. The method of claim 7, wherein determining the critical data elements based at least in part on preliminary analytics comprising correlation analysis further comprises defining a relationship between the critical data elements.

12. The method of claim 11, wherein determining the critical data elements based at least in part on preliminary analytics comprising correlation analysis further comprises narrowing a number of the critical data elements.

13. The method of claim 7, wherein determining the critical data elements based at least in part on preliminary analytics comprising regression analysis further comprises further comprises determining what other critical data elements exist based a degree of match between two data elements or increase or decrease in one critical data element with respect to an increase or decrease in another critical data element.

14. The method of claim 1, wherein determining the critical data elements further comprises determining the critical data elements based at least on part on subject matter expert input.

15. The method of claim 1, wherein building the data quality rules further comprises building a plurality of different sets of data quality rules, each set comprising a different number of data quality rules.

16. The method of claim 15, wherein building the data quality rules further comprises building the data quality rules based at least in part on the calculated data dimensions to determine what types of data quality rules are needed around the calculated data dimensions.

17. The method of claim 16, further comprising determining a cost for each set of data quality rules as a function of a number of rules in each set, a complexity of the rules in each set, and an interdependency of the rules in each set.

18. The method of claim 17, further comprising determining an overall benefit of each set of data quality rules based on a benefit of having a correct database less a sum of a cost of creating the data quality rules and a cost of poor data quality.

19. The method of claim 18, further comprising selecting one of the plurality of sets of data quality rules that optimizes the overall benefit.

20. The method of claim 19, wherein selecting one of the plurality of sets of data quality rules further comprises optimizing the overall benefit using non-parametric utility-based methodologies consisting of at least one of a decision tree method and a neural network method.

21. The method of claim 19, wherein selecting one of the plurality of sets of data quality rules further comprises selecting one of the plurality of sets of data quality rules that optimizes the overall benefit using parametric methodologies consisting of at least one of a linear regression method and a correlation method.

22. The method of claim 1, wherein monitoring the quality of the data further comprises monitoring the quality of the data using statistical process control methods.

23. The method of claim 22, wherein monitoring the quality of the data further comprises determining whether or not the quality of the data meets pre-determined expectations or is below the pre-determined expectations at a point of use of the data.

24. The method of claim 1, wherein identifying critical data elements that produce the pre-defined high number of outliers further comprises identifying the critical data elements that produce the pre-defined high number of outliers based on statistical process control methodologies using Pareto distribution for the outliers.

25. The method of claim 1, wherein identifying critical data elements that produce the pre-defined high number of outliers further comprises performing a correlation analysis and generating a scatter plot matrix of the correlation analysis.

26. The method of claim 25, wherein performing the correlation analysis and generating the scatter plot matrix further comprises identifying a degree of correlation between the critical data elements based on the scatter plot matrix.

27. The method of claim 1, wherein identifying a cause for the outliers further comprises performing a root cause analysis to determine which critical data elements cause the outliers.

28. The method of claim 1, wherein identifying a cause for the outliers further comprise performing advanced analytics using fishbone diagrams to capture the cause for the outliers.

29. A system for data quality assessment, comprising:

a microprocessor coupled to memory, wherein the microprocessor is programmed for;

collecting samples of data elements from a database storing a population of data elements representing attributes of each of a plurality of different financial transactions;

determining critical data elements from the collected samples of data elements;

building data quality rules and calculating data dimensions for the critical data elements;

monitoring a quality of data within the critical data elements for different data quality dimensions;

identifying critical data elements that produce a pre-defined high number of outliers; and

identifying causes for the outliers.

30. A non-transitory computer-readable storage medium with an executable program for assessing data quality stored thereon, wherein the program instructs a microprocessor to perform the steps of:

collecting samples of data elements from a database storing a population of data elements representing attributes of each of a plurality of different financial transactions;

determining critical data elements from the collected samples of data elements;

building data quality rules and calculating data dimensions for the critical data elements;

monitoring a quality of data within the critical data elements for different data quality dimensions;

identifying critical data elements that produce a pre-defined high number of outliers; and

identifying causes for the outliers.