Process and method for data assurance management by applying data assurance metrics

• US 8,843,487 B2
• Filed: 08/18/2010
• Issued: 09/23/2014
• Est. Priority Date: 08/18/2009
• Status: Expired due to Fees

• Current Assignee: Black Oak Partners LLC
• Original Assignee: Black Oak Partners LLC
• Inventors: McGraw, Thomas Rickett, Burks, Larry Ward, Kolo, Brian
• Primary Examiner(s): NGUYEN, CAM LINH T
• Application Number:
  US13/391,457
• Publication Number:
  US 20120158678A1
• Time in Patent Office: 1,497 Days
• Field of Search: 707/687, 707/999.101, 707/694, 707/736, 707/748, 707/803
• US Class Current: 707/736
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

The present invention relates generally to methods, software and systems for measuring and valuing the quality of information and data, where such measurements and values are made and processed by implementing objectively defined, measurable, comparable and repeatable dimensions using software and complex computers. The embodiments include processes, systems and method for identifying optimal scores of the data dimension. The invention further includes processes, systems and method for data filtering to improve the overall data quality of a data source. Finally, the invention further includes processes, systems and method for data quality assurance of groups of rows of a database.

Claims

1. A data assurance management method comprising:

selecting a plurality of data elements based on user requirements;

conducting a statistical random sampling of the plurality of data elements;

scoring, by one or more processors, the statistical random sampling to determine absolute and relative value of one or more data metrics, wherein the one or more data metrics are measures of data quality dimensions, wherein the data quality dimensions are characteristics of the plurality of data elements;

determining one or more frontier data points;

selecting an optimal data aggregation based on the one or more frontier data points;

applying the optimal data aggregation to the statistical random sample; and

rank ordering the aggregated data to create an output database from resultant data where at least a portion of less relevant data is eliminated in the output database.

2. The method of claim 1, wherein the plurality of data elements are units of information each having a unique meaning and distinct value.

3. The method of claim 1, wherein the plurality of data elements each comprise a name of the data element, a definition and an enumerated value.

4. The method of claim 1, wherein the statistical random sampling is a portion of an entire population selected to represent the entire population.

5. The method of claim 1, wherein the statistical random sampling is a subpopulation of interest.

6. The method of claim 1, further comprising ordering the statistical random sampling.

7. The method of claim 1, wherein the one or more data metrics are a measure of data quality dimension used in the scoring, wherein the data quality dimension is a characteristic of the statistical random sampling.

8. The method of claim 7, wherein the data quality dimensions are selected from the group consisting of:

accuracy, redundancy/uniqueness, velocity, acceleration, completeness, measure, timeliness, coverage, consistency, availability, read time, write time, propagation time, and combinations thereof.

9. The method of claim 8, wherein the selected data quality dimensions are predetermined.

10. The method of claim 9, wherein the scoring further comprises applying a first data quality dimension, applying a first score to each member of the statistical random sampling, applying a subsequent data quality dimension, applying a subsequent score to each member of the statistical random sampling, and repeating until all predetermined data quality dimensions are applied.

11. The method of claim 1, further comprising performing a multivariate optimization.

12. The method of claim 1, wherein the one or more frontier data points are incomparable data elements resulting from combining data sources.

13. The method of claim 1, wherein the optimal data aggregation is predetermined.

14. The method of claim 1, wherein the optimal data aggregation determines relative importance of each data quality dimension.

15. The method of claim 1, wherein the applying the optimal data aggregation comprises determining if the data is entered into an integrated database, and entering the resultant data in the integrated database if the resultant data is unique or if a rules engine selects the resultant data as an optimal data set.