Systems and methods for providing data quality management

• US 10,185,728 B2
• Filed: 12/19/2017
• Issued: 01/22/2019
• Est. Priority Date: 12/19/2016
• Status: Active Grant

• Current Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Original Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Inventors: Nath, Yatindra, Garg, Ankur, Tiwari, Rajeev, Vrat, Pranav, Mohanty, Amit
• Primary Examiner(s): Cao, Phuong Thao
• Application Number:
  US15/847,674
• Publication Number:
  US 20180173733A1
• Time in Patent Office: 399 Days
• Field of Search: 707692
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/025 Extracting rules from data

Abstract

A system for providing data quality management may include a processor configured to execute instructions to: extract a plurality of first data elements from a data source; generate a data profile based on the first data elements; automatically create a first set of rules based on the first data elements and the data profile, the first set of rules assessing data quality according to a threshold; generate a second set of rules based on the first data elements and the first set of rules; extract a plurality of second data elements; assess the second data elements based on a comparison of the second data elements to the second set of rules; detect defects based on the comparison; analyze data quality according to the detected defects; and transmit signals representing the data quality analysis to a client device for display to a user.

Claims

1. A system for providing data quality management, the system comprising:

a memory storing instructions; and

a processor connected to a network and configured to execute the instructions to;

extract a plurality of first data elements from a data source;

generate a data profile based on the first data elements;

automatically create a first set of rules based on the first data elements and the data profile, the first set of rules assessing data quality according to a threshold;

generate a second set of rules based on the first data elements and the first set of rules;

extract a plurality of second data elements;

assess the second data elements based on a comparison of the second data elements to the second set of rules;

receive a request from a user to adjust settings, the adjusted settings influencing identification of a node with a concentration of defects;

detect one or more defects based on the comparison and the user request, at least one of the detected one or more defects including an event;

cluster the assessed data elements into multiple segments according to the detected one or more defects in the assessed data elements, the multiple segments corresponding to data quality scores;

analyze, using a decision tree algorithm, data quality according to the detected one or more defects to determine a pocket of defect concentration and aggregate data quality; and

transmit signals representing the data quality scores and the data quality analysis to a client device for display to a user.

2. The system of claim 1, wherein extracting the plurality of first data elements from the data source comprises parsing the first data elements according to at least one of an alphanumeric identifier or a data list from the data source.

3. The system of claim 1, wherein the data profile comprises at least one of organizational information, authentication timestamp information, network node location, network node preference, access point location, or access point preference.

4. The system of claim 1, wherein generating the second set of rules comprises:

implementing a machine learning algorithm to learn from the first data elements and the first set of rules; and

generating the second set of rules based on the learning.

5. The system of claim 1, wherein generating the second set of rules comprises generating the second set of rules based on a user input for adjusting the first set of rules.

6. The system of claim 1, wherein defining a conditional rule comprises identifying at least one of subpopulations or contingent relationships between the first data elements.

7. The system of claim 1, wherein the second set of rules comprises a framework criteria comprising at least one of a support value, a confidence value, or a lift ratio.

8. The system of claim 1, wherein the first set of rules comprises at least one of a null rule, a range rule, a uniqueness rule, a valid value rule, a format rule, a conditional rule, or a consistency rule.

9. A method for providing data quality management, the method comprising the following operations performed by a processor:

extracting a plurality of first data elements from a data source;

generating a data profile based on the first data elements;

automatically creating a first set of rules based on the first data elements and the data profile, the first set of rules assessing data quality according to a threshold;

generating a second set of rules based on the first data elements and the first set of rules;

extracting a plurality of second data elements;

assessing the second data elements based on a comparison of the second data elements to the second set of rules;

receiving a request from a user to adjust settings, the adjusted settings influencing identification of nodes with a concentration of defects;

detecting one or more defects based on the comparison and the user request, at least one of the detected one or more defects including an event;

clustering the assessed data elements into multiple segments according to the detected one or more defects in the assessed data elements, the multiple segments corresponding to data quality scores;

analyzing, using a decision tree algorithm, data quality according to the detected one or more defects to determine a pocket of defect concentration and aggregate data quality; and

transmitting signals representing the data quality scores and the data quality analysis to a client device for display to a user.

10. The method of claim 9, wherein extracting the plurality of first data elements from the data source comprises:

parsing the plurality of first data elements according to at least one of an alphanumeric identifier or a data list from the data source.

11. The method of claim 9, wherein the data profile comprises at least one of organizational information, authentication timestamp information, network node location, network node preference, access point location, or access point preference.

12. The method of claim 9, wherein generating the second set of rules comprises:

implementing a machine learning algorithm to learn from the first data elements and the first set of rules; and

generating the second set of rules based on the learning.

13. The method of claim 9, wherein generating the second set of rules comprises:

generating the second set of rules based on a user input for adjusting the first set of rules.

14. The method of claim 9, further comprising:

defining a conditional rule by identifying at least one of subpopulations or contingent relationships between the first data elements.

15. The method of claim 9, wherein the second set of rules comprises a framework criteria comprising at least one of a support value, a confidence value, or a lift ratio.

16. The method of claim 9, wherein the first set of rules comprises at least one of a null rule, a range rule, a uniqueness rule, a valid value rule, a format rule, a conditional rule, or a consistency rule.

17. A non-transitory computer-readable medium storing instructions executable by a processor to perform a method for providing data quality management, the method comprising:

extracting a plurality of first data elements from a data source;

generating a data profile based on the first data elements;

automatically creating a first set of rules based on the first data elements and the data profile, the first set of rules assessing data quality according to a threshold;

generating a second set of rules based on the first data elements and the first set of rules;

extracting a plurality of second data elements;

assessing the second data elements based on a comparison of the second data elements to the second set of rules;

receiving a request from a user to adjust settings, the adjusted settings influencing identification of nodes with a concentration of defects;

detecting one or more defects based on the comparison;

clustering the assessed data elements into multiple segments according to the detected one or more defects in the assessed data elements, the multiple segments corresponding to data quality scores;

analyzing, using a decision tree algorithm, data quality according to the detected one or more defects to determine a pocket of defect concentration and aggregate data quality; and

transmitting signals representing the data quality scores and the data quality analysis to a client device for display to a user.

18. The non-transitory computer-readable medium of claim 17, wherein extracting the plurality of first data elements from the data source comprises:

parsing the plurality of first data elements according to at least one of an alphanumeric identifier or a data list from the data source.

19. The non-transitory computer-readable medium of claim 17, wherein the data profile comprises at least one of organizational information, authentication timestamp information, network node location, network node preference, access point location, or access point preference.

20. The non-transitory computer-readable medium of claim 17, wherein generating the second set of rules comprises:

implementing a machine learning algorithm to learn from the first data elements and the first set of rules; and

generating the second set of rules based on the learning.