PROCESS AND METHOD FOR DATA ASSURANCE MANAGEMENT BY APPLYING DATA ASSURANCE METRICS

• US 20140324801A1
• Filed: 05/01/2014
• Published: 10/30/2014
• Est. Priority Date: 08/18/2009
• Status: Abandoned Application

• Current Assignee: Black Oak Partners LLC
• Original Assignee: Black Oak Partners LLC
• Inventors: MCGRAW, Thomas Rickett, BURKS, Larry Ward, KOLO, Brian
• Application Number:
  US14/267,106
• Publication Number:
  US 20140324801A1
• Time in Patent Office: Days
• US Class Current: 707/694
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

The present invention relates generally to methods, software and systems for measuring and valuing the quality of information and data, where such measurements and values are made and processed by implementing objectively defined, measurable, comparable and repeatable dimensions using software and complex computers. The embodiments include processes, systems and method for identifying optimal scores of the data dimension. The invention further includes processes, systems and method for data filtering to improve the overall data quality of a data source. Finally, the invention further includes processes, systems and method for data quality assurance of groups of rows of a database.

Claims

1. A data filtering method comprising:

receiving data from a database;

determining if the data is valid by utilizing rules in a data hygiene engine, wherein the rules are;

(1) at least one rule selected from the group consisting of;

field rules, row rules, groups of rows rules, and combinations thereof, and

(2) at least one data assurance management metrics rules, wherein the data assurance management metrics rules are data quality metric rules used to analyze groups of rows based on user requirements, wherein data quality metrics are calculated for groups of rows and compared to a threshold;

filtering invalid data;

outputting a production database without the invalid data.

2. The method of claim 1, wherein the field rules are related to only one field.

3. The method of claim 1, wherein the field rules are field type or field value ranges.

4. The method of claim 1, wherein the row rules are rules to intermix two different fields.

5. The method of claim 1, wherein the groups of row rules group rows, calculate metrics for each group of rows, and comparing to a threshold.

6. The method of claim 5, wherein an entire group is eliminated if the calculated metric is below the threshold.

7. The method of claim 5, wherein the threshold is predetermined by a user.

8. The method of claim 1, wherein a data assurance management metric is used to group the data into multiple groups to complete a score for the entire dataset.

9. The method of claim 8, further comprising comparing the score to a threshold and eliminating all groups that do not meet the threshold.

10. The method of claim 1, wherein a data assurance management metric is used to group the data into multiple groups and compute a completeness score for the entire dataset.

11. The method of claim 10, further comprising comparing the completeness score to a minimum threshold and eliminating all groups that do not meet the threshold.

12. The method of claim 1, wherein a plurality of data assurance management metrics are used together to group the data into multiple groups and compute a score for the entire dataset.

13. The method of claim 12, further comprising comparing the completeness score to a threshold and eliminating all groups that do not meet the threshold.

14. The method of claim 13, wherein the threshold is determined by a rule related to the plurality of data assurance management metrics.

15. The method of claim 1, wherein an acceptable range of values or numbers is provided to the data hygiene engine.

16. The method of claim 1, wherein invalid data is subject to further review before being filtered.

17. The method of claim 1, wherein the invalid data is modified to create valid data that is then considered valid.

18. The method of claim 1, further comprising running a search engine query coupled to a threshold for the at least one data assurance management metrics.

19. The method of claim 18, further comprising setting reliability thresholds.

20. The method of claim 18, further comprising setting confidence thresholds.