Data quality administration framework

• US 8,606,762 B2
• Filed: 09/16/2008
• Issued: 12/10/2013
• Est. Priority Date: 09/16/2008
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP AG (SAP SE)
• Inventors: Schreck, Stefan, Meyer, Arno
• Primary Examiner(s): Beausoliel, Jr., Robert
• Application Number:
  US12/211,369
• Publication Number:
  US 20100070893A1
• Time in Patent Office: 1,911 Days
• Field of Search: 707/692, 707/602, 707/705, 707/737, 707/769, 707/792, 707/803
• US Class Current: 707/692
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance

Abstract

A method of data quality management including selecting a task group wherein the task group comprises at least one data validation sub-task. In one embodiment, the method may also include arranging at least a portion of a master data into a validation group, wherein the validation group is associated with the task group. In various embodiments, the method may include, for each data validation sub-task, determining a sub-task type, validating the validation group as directed by the data validation sub-task, and calculating a set of key performance indicators (KPIs) associated with the data validation sub-task.

Claims

1. A method of data quality management comprising:

selecting, by a processor of a data quality manager device, a task group wherein the task group comprises at least one data validation sub-task including a set of configurable parameters for controlling execution of the at least one data validation sub-task;

arranging at least a portion of a master data, stored by a memory of a database device, into a validation group, wherein the validation group is associated with the task group; and

for each data validation sub-task;

determining a sub-task type of the data validation sub-task,

validating the validation sub-task based upon the sub-task type of the data validation sub-task, and

calculating a set of key performance indicators (KPIs) associated with the data validation sub-task.

2. The method of claim 1 further comprising selecting a sub-task type from a group consisting of:

a contact information validation sub-task type;

a content validation sub-task type; and

a duplicate check sub-task type.

3. The method of claim 1 further comprising determining a set of preconfigured settings for each data validation sub-task;

and

wherein the set of preconfigured settings comprises;

a sub-task type for the data validation sub-task;

whether or not the data validation sub-task is to be performed using an external validation service provider; and

at least one of the following;

file format used to export information to the external validation service provider, and

a file format used to import information from the external validation service provider a list of key performance indicators to be calculated as a result of the data validation sub-task.

4. The method of claim 1 wherein the validation group comprises a database of business records;

and

wherein validating comprises;

determining, based in part upon the data validation sub-task, a set of conditions that indicate the quality of a record under test,

testing at least a portion of each record to determine if the record conforms to the set of conditions, and

if a record fails testing, indicating that the record is faulty.

5. The method of claim 4 wherein validating further comprises:

if a record fails the test, indicating at least one remedial measure to take to correct the record.

6. The method of claim 1 wherein the validation group comprises a database of business records;

and

wherein validating comprises, if the data validation sub-task type comprises a contact information validation sub-task type, for each record of the validation group;

validating a syntax of a set of contact information comprised in the record, and

validating a spelling of the set of contact information.

7. The method of claim 1 wherein the validation group comprises a database of business records;

and

wherein validating comprises, if the data validation sub-task type comprises a content validation sub-task type, for each record of the validation group;

validating an existence of a business partner represented by the record,

validating that the business partner has a location at a set of contact information comprised in the record, and

validating a selected set of other information comprised in the record.

8. The method of claim 1 wherein the validation group comprises a database of business records;

and

wherein validating comprises, if the data validation sub-task type comprises a duplication checking sub-task type, for each record of the validation group;

determining if a record is substantially equivalent to any other record in the validation group,

if so, generating a cleansing case, indicating that the record and the record'"'"'s substantial equivalent record(s) are effectively duplicates, and

automatically merging the effectively duplicate records to a single record.

9. The method of claim 1 wherein validating comprises:

determining, based upon a sub-task type, whether or not the validation is to be performed by an external validation service provider; and

if so,

exporting data from the validation group to a predetermined file,

initiating the external validation service provider to validate and update the data comprised in the exported data file,

acquiring a validated file, produced by the external validation service provider, that comprises any updated portions of the data, and

merging, from the file, at least the updated portions of the data into the validation group.

10. The method of claim 9 wherein merging comprises:

importing a file that comprises a plurality of records, each record comprising an action code field indicating an action to be taken when merging the file with the validation group,

wherein the action comprises an action selected from a group comprising;

no action, create a record, change a record, delete a record; and

if the sub-task associated with the file comprises a duplicate check sub-task type, importing a file that comprises for each record a cleansing case field indicating which records are substantially equivalent.

11. The method of claim 1 wherein calculating comprises:

calculating a number of records in the validation group;

determining a data verification sub-task performed;

calculating a number of records marked as needing correction; and

calculating a number of records corrected.

12. The method of claim 1 further comprising:

determining if any data validation sub-tasks are prerequisites for other data validation sub-tasks;

if so, arranging the data validation sub-tasks such that the prerequisite data validation sub-tasks are performed prior to their respective dependent sub-tasks;

determining if any data validation sub-tasks may be performed in parallel; and

if so, arranging the data validation sub-tasks to facilitate the parallel performance of the sub-tasks.

13. A data quality manager comprising:

a memory configured to;

store at least a portion of a validation group comprising at least a portion of a set of master data, and

store at least a portion of a task group, associated with the validation group, indicating a set of data validation sub-tasks to be performed using the validation group, each data validation sub-task including a set of configurable parameters for controlling execution of each data validation sub-task;

a processor configured to;

determine, for each data validation sub-task which validation engine to use to validate the validation group according to a sub-task type of the respective data validation sub-task;

at least one validation engine configured to;

for at least a subset of the data validation sub-tasks, validate the validation group according to the data validation sub-task; and

a key performance indicator (KPI) calculator configured to;

calculate a set of KPIs associated with the data validation sub-task, and

provide the KPIs to a data validation user interface.

14. The data quality manager of claim 13 further comprising a data validation user interface configured to provide a first user interface element with which a user can select a task group to use for data quality management;

and display the results of the data validation sub-tasks of the task group.

15. The data quality manager of claim 13 wherein each data validation sub-task comprises a sub-task type from a group consisting of:

a contact information validation sub-task type;

a content validation sub-task type; and

a duplicate check sub-task type.

16. The data quality manager of claim 13 wherein the validation group comprises at least a portion of a database of records;

and

wherein the at least one validation engine comprises a contact information validation engine specifically configured to provide contact information validation; and

wherein the contact information validation engine is configured to, for each record of the validation group;

validate a syntax of a set of contact information comprised in the record, and validate the spelling of a set of contact information.

17. The data quality manager of claim 13 wherein the validation group comprises at least a portion of a database of records;

and

wherein the at least one validation engine comprises a content validation engine specifically configured to provide content validation; and

wherein the content validation engine is configured to;

validate an existence of a business partner represented by the record, validate that the business partner has a location at a set of contact information comprised in the record, and validate a selected set of other information comprised in the record.

18. The data quality manager of claim 13 wherein the validation group comprises at least a portion of a database of records;

and

wherein the at least one validation engine comprises a duplication checking validation engine specifically configured to provide duplication validation; and

wherein the duplication checking validation engine is configured to;

determine if a record is substantially equivalent to any other record in the validation group, if so, generate a cleansing case, indicating that the record and the record'"'"'s substantial equivalent record(s) are effectively duplicates, and automatically merge the effectively duplicate records to a single record.

19. The data quality manager of claim 13 wherein the processor is configured to determine whether or not the validation is to be performed by an external validation service provider;

wherein the data quality manager comprises a data importer/exporter configured to;

export data from the validation group to an export file,

merge, from an import file produced by the external validation service provider, at least the updated portions of the data into the validation group; and

wherein the processor is configured to;

cause the external validation service provider to validate and update the data comprised in the export file.

20. The data quality manager of claim 19 wherein data importer/exporter is configured to:

import an import file that comprises a plurality of records, each record comprising an action code field indicating an action to be taken when merging the file with the validation group, wherein the action comprises an action selected from a group comprising;

no action, create a record, change a record, delete a record; and

if the sub-task type associated with the file comprises a duplicate check sub-task type, import an import file that comprises for each record a cleansing case field indicating which records are substantially equivalent.

21. The data quality manager of claim 13 further comprising a sub-task arranger configured to:

determine if any data validation sub-tasks are prerequisites for other data validation sub-tasks;

if so, arrange the data validation sub-tasks such that the prerequisite data validation sub-tasks are performed prior to their respective dependent sub-tasks;

determine if any data validation sub-tasks may be performed in parallel; and

if so, arrange the data validation sub-tasks to facilitate the parallel performance of the sub-tasks.

22. A machine readable medium comprising instructions being tangibly and non-transitorily embodied and capable of being executed by at least one machine, wherein the instructions, when executed, cause the machine to:

select a task group wherein the task group comprises at least one data validation sub-task including a set of configurable parameters for controlling execution of the at least one data validation sub-task;

arrange at least a portion of a master data into a validation group, wherein the validation group is associated with the task group; and

for each data validation sub-task;

determine a sub-task type of the data validation sub-task,

validate the validation sub-task based upon the sub-task type of the data validation sub-task, and

calculate a set of key performance indicators (KPIs) associated with the data validation sub-task.