Data comparison system

• US 9,122,732 B2
• Filed: 07/20/2010
• Issued: 09/01/2015
• Est. Priority Date: 08/06/2009
• Status: Active Grant

• Current Assignee: Accenture Global Services Limited (Accenture PLC)
• Original Assignee: Accenture Global Services Limited (Accenture PLC)
• Inventors: Pong, Geoffrey, Hui, Anita
• Primary Examiner(s): Uddin, Mohammed R
• Application Number:
  US12/839,975
• Publication Number:
  US 20110035371A1
• Time in Patent Office: 1,869 Days
• Field of Search: 707/758, 707/690, 707/692, 707/701, 707/999.101
• US Class Current: 1/1
• CPC Class Codes: G06F 16/214 Database migration support
• CPC Class Codes: G06F 16/2308 Concurrency control transaction processing G06F9/466
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems

Abstract

A data comparison system is described. The system may include a memory, an interface, and a processor. The memory may store a compare data structure containing multiple data item pairs, each pair including a legacy data item of a legacy dataset and a corresponding new data item of a new dataset, and a tolerance associated with each data item pair. The processor may receive the compare data structure and the associated tolerances. The processor may call a compare data subroutine to compare each data item pair in accordance with the associated tolerance if the data items are fields. Otherwise, the processor may recursively call the compare data subroutine for each record the data items until the data item are fields. The processor may then compare data items in accordance with the associated tolerance.

Claims

1. A computer-implemented method for performing a tolerance based comparison between a legacy data store and a new data store, the method comprising:

receiving, by a processor from a device of a user, a compare data structure comprising a plurality of data item pairs, each data item pair identifying a legacy data item of a legacy dataset and a new data item of a new dataset,

wherein each data item pair comprises a data type of a table data type, a flat structure data type, a deep structure data type, or a field data type, and

wherein the table data type, the flat structure data type and the deep structure data type each comprise a plurality of records;

receiving, by the processor from the device of the user, a plurality of tolerances, each tolerance being associated with one of the data item pairs and indicative of an acceptable difference between the data item pair according to the data type of the data item pair;

recursively comparing, by the processor, each data item pair of the plurality of data item pairs wherein recursively comparing comprises determining the data type of each data item pair, and;

when the data item pair is a determined to be the table data type, calling a compare subroutine for each record in each table of the data item pair to form new data item pairs to compare;

when the data item pair is determined to be the flat structure data type or the deep structure data type, calling the compare subroutine for each record in the flat structure data or the deep structure of the data item pair to form new data item pairs to compare;

when the data item pair is determined to not be one of the table data type, the flat structure data type, the deep structure data type, or the field data type, writing a log entry indicating that the data item pair is an unknown data type;

determining, by the processor, that each of one or more of the plurality of data item pairs being compared comprises the field data type;

identifying a subset of the plurality of data item pairs comprising the one or more of the plurality of data item pairs determined to be of the field data type; and

for each of the one or more of the plurality of data item pairs determined to be of the subset of the plurality of data item pairs;

checking, by the processor, each legacy data item in relation to each new data item of each data item pair in accordance with the associated tolerance; and

assigning, by the processor, a category among a plurality of categories for each data item pair determined to be of the subset based on the difference of each data item pair within the tolerance associated with each data item pair, wherein the plurality of categories comprises

an exact match category,

a within tolerance category, and

an outside of tolerance category;

transforming, by the processor, a result of the checking and assigning into a report, wherein the report describes

a percentage of the data item pairs assigned the exact match category,

a percentage of the data item pairs assigned the within tolerance category, and

a percentage of the data item pairs assigned the outside of tolerance category; and

providing, by the processor to the device of the user, the report.

2. The computer-implemented method of claim 1 further comprising:

transforming, by the processor, the result of the checking into the report, wherein the report describes

a percentage of the legacy and new data items which were an exact match,

a percentage of the legacy and new data items which were within the associated tolerance, and

a percentage of the legacy and new data items which were outside the associated tolerance; and

providing, by the processor to the device of the user, the report.

3. The computer-implemented method of claim 2 wherein the report comprises a plurality of field-level statistics and a plurality of record level statistics.

4. The computer-implemented method of claim 3 wherein the report groups the plurality of field-level statistics based on a field in the plurality of fields.

5. The computer-implemented method of claim 1 wherein receiving the plurality of tolerances further comprises:

providing, by the processor to the device of the user, a user interface for identifying the plurality of tolerances and the data item pair associated with each tolerance; and

receiving, from the device of the user, the plurality of tolerances, each tolerance being associated with one of the data item pairs, wherein each tolerance is associated with a threshold variance value between the legacy data item and the new data item in the associated data pair.

6. The computer-implemented method of claim 1 wherein at least one data item comprises a data type of a character data type and the associated tolerance comprises at least one of a case sensitive tolerance and a space sensitive tolerance.

7. The computer-implemented method of claim 1 wherein at least one data item comprises a data type of a date data type and the associated tolerance comprises a number of days difference tolerance.

8. The computer-implemented method of claim 1 wherein the legacy dataset and the new dataset comprise a plurality of invoices.

9. The computer-implemented method of claim 1 further comprising:

providing, to the device of the user, a user interface for identifying which of the data item pairs to compare; and

receiving, by the processor from the device of the user, a selection of data item pairs to compare.

10. The computer-implemented method of claim 9 wherein checking, by the processor, each legacy data item and each new data item of each data item pair in accordance with the associated tolerance if the data type of each data item pair comprises the field data type further comprises

comparing, by the processor, each legacy data item and each new data item of each data item pair in accordance with the associated tolerance if the data type of each data item pair comprises the field data type and if the data item pair was provided, to the device of the user, a user interface for identifying which of the data item pairs to compare.

11. A computer-implemented method for performing a tolerance based comparison between a legacy data store and a new data store, the method comprising:

receiving, by a processor from a device of a user, a compare data structure,

wherein the compare data structure describes a plurality of mappings between a first plurality of data items of a first dataset and a second plurality of data items of a second dataset, and

wherein the data items comprise at least one of

a table,

a field,

a deep structure, or

a flat structure;

populating, by the processor, the compare data structure with a first plurality of data items from the first dataset and the second plurality of data items from the second dataset;

receiving, by the processor, a plurality of tolerances, each tolerance being associated with one of the mappings between the first plurality of data items and the second plurality of data items, wherein each tolerance is associated with a threshold variance value for each mapping of the plurality of mappings and indicative of an acceptable difference between the data item pair according to the data type of the data item pair;

recursively comparing, by the processor, each data item pair of the plurality of data item pairs wherein recursively comparing comprises determining the data type of each data item pair, and;

when the data item pair is a determined to be the table data type, calling a compare subroutine for each record in each table of the data item pair to form new data item pairs to compare;

when the data item pair is determined to be the flat structure data type or the deep structure data type, calling the compare subroutine for each record in the flat structure data or the deep structure of the data item pair to form new data item pairs to compare;

when the data item pair is determined to not be one of the table data type, the flat structure data type, the deep structure data type, or the field data type, writing a log entry indicating that the data item pair is an unknown data type;

determining, by the processor, that each of one or more of the first plurality of data items and of the second plurality of data items comprises the field data type;

identifying a subset of the first plurality of data items comprising the one or more of the first plurality of data items determined to be of the field data type and

identifying a subset of the second plurality of data items comprising the one or more of the second plurality of data items determined to be of the field data type;

for each of the one or more of the first and second plurality of data items determined to be of the subset of the plurality of data item pairs;

checking, by the processor, the data items of the subset of the first plurality of data items in the compare data structure with the data items of the subset of the second plurality of data items in the compare structure in accordance with the plurality of mappings and the plurality of tolerances; and

assigning, by the processor, a category among a plurality of categories for each data item pair determined to be of the subset based on the difference of each data item pair within the tolerance associated with each data item pair, wherein the plurality of categories comprises

an exact match category,

a within tolerance category, and

an outside of tolerance category;

transforming, by the processor, a result of the checking and assigning into a report, wherein the report describes

a percentage of the data items of the first and second plurality of data items assigned to the exact match category,

a percentage of the data items of the first and second plurality of data items assigned to the within tolerance category, and

a percentage of the data items of the first and second plurality of data items assigned to the outside of the tolerance category; and

providing, by the processor to the device of the user, the report.

12. The computer-implemented method of claim 11 wherein the report comprises a plurality of field-level statistics and a plurality of record level statistics.

13. The computer-implemented method of claim 12 wherein the report groups the plurality of field-level statistics based on a field in the plurality of fields.

14. The computer-implemented method of claim 11 wherein at least one data item comprises a data type of a character data type and the associated tolerance comprises at least one of a case sensitive tolerance and a space sensitive tolerance.

15. The computer-implemented method of claim 11 wherein at least one data item comprises a data type of a date data type and the associated tolerance comprises a number of days difference tolerance.

16. The computer-implemented method of claim 11 wherein the first dataset and the second dataset comprise a plurality of invoices.

17. The computer-implemented method of claim 11 further comprising:

providing, to the device of the user, a user interface for identifying which of the mappings to check; and

receiving, by the processor from the device of the user, a selection of mappings to check.

18. The computer-implemented method of claim 17 wherein checking, by the processor, the first plurality of data items in the compare data structure with the second plurality of data items in the compare structure in accordance with the plurality of mappings and the plurality of tolerances if each mapping was selected by the user.

19. A data comparison system, the system comprising:

a memory to store a compare data structure comprising

a plurality of data item pairs, each data item pair identifying a legacy data item of a legacy dataset and a new data item of a new dataset,

wherein each data item pair comprises a data type of

a field data type,

a table data type,

a flat structure data type, or

a deep structure data type,

and wherein the table data type, the flat structure data type and the deep structure data type each comprise

a plurality of records, and

a plurality of tolerances, each tolerance being associated with one of the data item pairs and indicative of an acceptable difference between the data item pair according to the data type of the data item pair;

an interface operatively connected to the memory, the interface operative to communicate with a device of a user; and

a processor operatively connected to the memory and the interface, the processor operative to

receive, from the device of the user via the interface, the compare data structure and the plurality of tolerances,

recursively call a compare data subroutine to compare each legacy data item and each new data item identified by each data item pair wherein recursively comparing comprises determining the data type of each data item pair, and;

when the data item pair is a determined to be the table data type, calling a compare subroutine for each record in each table of the data item pair to form new data item pairs to compare;

when the data item pair is determined to be the flat structure data type or the deep structure data type, calling the compare subroutine for each record in the flat structure data or the deep structure of the data item pair to form new data item pairs to compare;

when the data item pair is determined to not be one of the table data type, the flat structure data type, the deep structure data type, or the field data type, writing a log entry indicating that the data item pair is an unknown data type;

determine that each of one or more of the data item pairs being compared comprises the field data type,

identify a subset of the data item pairs comprising the one or more of the plurality of data item pairs determined to be of the field data type, and

for each of the one or more of the data item pairs determined to be of the subset of the data item pairs;

check each legacy data item and each new data item in accordance with the associated tolerance if the data type of each data item pair in accordance with the associated tolerance; and

assign a category among a plurality of categories for each data item pair determined to be of the subset based on the difference of each data item pair within the tolerance associated with each data item pair, wherein the plurality of categories comprises

an exact match category,

a within tolerance category, and

an outside of tolerance category.

20. The system of claim 19, wherein the report describes

a percentage of a first and second data items which were an exact match,

a percentage of the first and second data items which were within the associated tolerance, and

a percentage of the first and second data items which were outside the associated tolerance, and provide, to the device of the user via the interface, the report.

21. The system of claim 20 wherein the report comprises a plurality of field-level statistics and a plurality of record level statistics.

22. The system of claim 21 wherein the report groups the plurality of field-level statistics based on a field in the plurality of fields.

23. The system of claim 19 wherein the processor is further operative to

provide, to the device of the user via the interface, a user interface for identifying the plurality of tolerances and the data item pair associated with each tolerance, and

receive, from the device of the user via the interface, the plurality of tolerances, each tolerance being associated with one of the data item pairs and is associated with a threshold variance value between the legacy data item and the new data item in the associated data pair.

24. The system of claim 19 wherein the processor is further operative to

compare each legacy data item and each new data item to determine a difference between each legacy data item and each new data item if each legacy data item and each new data item comprise the field, and

assign one of the plurality of categories for each data item pair, the category assigned based on whether the difference of each data item pair is within the associated tolerance.

25. The system of claim 24 wherein each category comprises one of an exact match category, a within tolerance category and an outside of tolerance category.

26. The system of claim 25 wherein the processor is further operative to transform each category into the report, wherein the report describes a percentage of the data item pairs which assigned the exact match category, a percentage of the data item pairs which were assigned the within tolerance category, and a percentage of the data item pairs which were assigned the outside of tolerance category, and provide, to the device of the user via the interface, the report.

27. The system of claim 19 wherein at least one data item comprises a data type of a character data type and the associated tolerance comprises at least one of a case sensitive tolerance and a space sensitive tolerance.

28. The system of claim 19 wherein at least one data item comprises a data type of a date data type and the associated tolerance comprises a number of days difference tolerance.

29. The system of claim 19 wherein the legacy dataset and the new dataset comprise a plurality of invoices.

30. The system of claim 19 wherein the processor is further operative to

provide, to the device of the user via the interface, a user interface for identifying which of the data item pairs to check, and

receive, from the device of the user via the interface, a selection of data item pairs to check.

31. The system of claim 30 wherein the processor is further operative to check each legacy data item and each new data item in accordance with the associated tolerance if each legacy data item and each new data item comprise the field and if the data item pair was selected by the user.