Managing data integrity using a filter condition

• US 7,464,097 B2
• Filed: 08/16/2002
• Issued: 12/09/2008
• Est. Priority Date: 08/16/2002
• Status: Expired due to Fees

• Current Assignee: SAP SE
• Original Assignee: SAP AG (SAP SE)
• Inventors: Schmitt, Michael, Lutz, Steffen, Goetz, Martina
• Primary Examiner(s): Mofiz; Apu
• Assistant Examiner(s): DAYE, CHELCIE L
• Application Number:
  US10/222,757
• Publication Number:
  US 20040034699A1
• Time in Patent Office: 2,307 Days
• Field of Search: 707/100, 707/201, 707/203, 707/101, 711/165
• US Class Current: 1/1
• CPC Class Codes: H04L 41/00 Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
• CPC Class Codes: H04L 41/024 using relational databases for representation of network management data, e.g. managing via structured query language [SQL]
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

Techniques are provided to manage the integrity of data stored in two or more data management systems by detecting inconsistencies between the data management systems. The techniques identify missing records in one or more data management systems by comparing the records in the data management systems. A filter condition is used to identify the records to be compared. For records that exist in two or more data management systems, the techniques identify records that are not identical in the data management systems. A user checkpoint is provided between the identification of missing records and the identification of records that are not identical. The detected inconsistencies also may be corrected.

Claims

1. A computer-implemented method for managing data integrity, the method comprising:

identifying data objects stored in a first data management system that meet a filter condition, each data object including an identifier that uniquely identifies a data object from other data objects in the first data management system, a set of attributes, and attribute values that correspond to attributes in the attribute set, and the filter condition comprising at least one value of an attribute, the attribute corresponding to an attribute in the attribute set;

identifying data objects stored in a second data management system that meet the filter condition, each data object including an identifier that uniquely identifies a data object from other data objects in the second data management system, a set of attributes, and attribute values that correspond to attributes in the attribute set;

accessing data that identifies excluded data objects that are excluded from managing data integrity even when the excluded data objects meet the filter condition;

based on the accessed data, determining a subset of the data objects identified from the first data management system as meeting the filter condition that excludes the excluded data objects and a subset of the data objects identified from the second data management system as meeting the filter condition that excludes the excluded data objects;

comparing identifiers from the subset of data objects identified from the first data management system with identifiers from the subset of data objects identified from the second data management system to determine whether each data object in one of the data management systems has a corresponding data object in the other data management system;

storing, in electronic storage, comparison results information indicating results of the comparison of the identifiers from the subset of data objects identified from the first data management system with the identifiers from the subset of data objects identified from the second data management system, the comparison results information indicating whether each data object that is included in at least one of the subsets and meets the filter condition is stored in the first data management system, the second data management system, or both the first data management system and the second data management system;

accessing the comparison results information; and

managing the integrity of the identified data objects based on the accessed comparison results information such that each data object that is included in at least one of the subsets and meets the filter condition is included both in the first data management system and in the second data management system, managing the integrity comprising;

when a data object that is included in at least one of the subsets and meets the filter condition occurs only in the first data management system, sending the data object from the first data management system to the second data management system,

when a data object that is included in at least one of the subsets and meets the filter condition occurs only in the second data management system, sending the data object from the second data management system to the first data management system,

when a data object that is included in at least one of the subsets and meets the filter condition occurs in both the first data management system and in the second data management system, determining whether a first set of attribute values associated with the data object in the first data management system is equal to a second set of attribute values associated with the data object in the second data management system, and

when the first set of attribute values is not equal to the second set of attribute values, sending the data object from the first data management system to the second data management system.

2. The method of claim 1 wherein managing the integrity comprises when the first set of attribute values is not equal to the second set of attribute values, deleting the data object from the second data management system and sending the data object from the first data management system to the second data management system.

3. The method of claim 1 wherein the filter condition comprises a user-definable filter condition.

4. The method of claim 1 wherein the filter condition comprises one or more values for one or more attributes included in a data object.

5. The method of claim 1 wherein the filter condition comprises a data object identifier.

6. The method of claim 1 further comprising:

extracting, from the first data management system, only identifiers for a first group of data objects that include multiple attribute values, are included in the subset of the data objects identified from the first data management system, and meet the filter condition, extracting only identifiers comprising extracting identifiers for the first group of data objects without extracting other of the multiple attribute values,

extracting, from the second data management system, only identifiers for a second group of data objects that include multiple attribute values, are included in the subset of the data objects identified from the second data management system, and meet the filter condition, extracting only identifiers comprising extracting identifiers for the second group of data objects without extracting other of the multiple attribute values,

wherein comparing identifiers from the subset of data objects identified from the first data management system with identifiers from the subset of data objects identified from the second data management system to determine whether each data object in one of the data management systems has a corresponding data object in the other data management system comprises comparing only the extracted identifiers for the first group of data objects that meet the filter condition with the extracted identifiers for the second group of data objects that meet the filter condition.

7. The method of claim 6 further comprising:

providing a user with an indication of the results of the comparison of identifiers;

permitting the user to determine whether to proceed with managing the integrity of the identified data objects based on the indication of the results of the comparison of the identifiers; and

only after receiving a first indication from the user to proceed;

extracting, from the first data management system, the other of the multiple attribute values from the first group of data objects that meet the filter condition,

extracting, from the second data management system, the other of the multiple attribute values from the second group of data objects that meet the filter condition,

for data objects included in both the first group of data objects and the second group of data objects, comparing the extracted attribute values from the first data management system with the extracted attribute values from the second data management system;

providing the user with an indication of the results of the comparison of the extracted attribute values;

permitting the user to determine whether to proceed with managing the integrity of the identified data objects based on the indication of the results of the comparison of the extracted attribute values; and

only after receiving a second indication from the user to proceed, managing the integrity of the identified data objects.

8. The method of claim 1 wherein a data object comprises an object instance, and the identifier is an object identifier.

9. The method of claim 1 wherein a data object comprises a data record, and the identifier is a record identifier.

10. The method of claim 1 wherein a data object comprises a row in a database table, and the identifier is a row identifier.

11. The method of claim 1 wherein a data object comprises a data segment, and the identifier is a data segment identifier.

12. The method of claim 1 further comprising:

determining the number of data objects that occur in both the first data management system and the second data management system and meet the filter condition;

determining the number of data objects that occur in only the first data management system and meet the filter condition; and

determining the number of data objects that occur in only the second data management system and meet the filter condition.

13. The method of claim 12 further comprising:

presenting the number of data objects that occur in both the first data management system and the second data management system and meet the filter condition;

presenting the number of data objects that occur in only the first data management system and meet the filter condition; and

presenting the number of data objects that occur in only the second data management system and meet the filter condition.

14. The method of claim 1 further comprising:

extracting from the first data management system a first group of data objects that meet the filter condition; and

extracting from the second data management system a second group of data objects that meet the filter condition.

15. The method of claim 14 further comprising:

receiving from the user an indication of one or more data objects that are associated both with the first group of data objects and with the second group of data objects, each single data object having a first set of attribute values associated with the first group of data objects and a second set of attribute values associated with the second group of data objects; and

providing the user with an indication of whether the first set of attribute values is equal to the second set of attribute values.

16. The method of claim 1 further comprising transforming one or more values for a set of attributes associated with a data object such that the set of attributes associated with the data object in the first data management system may be compared with the set of attributes associated with the data object in the second data management system.

36. The method of claim 1 wherein the comparison results information includes a data integrity record for each data object that meets the filter condition.

37. The method of claim 36 wherein each data integrity record includes a data integrity identifier that uniquely identifies the data integrity record.

38. The method of claim 37 wherein each data integrity record further includes:

a field for a comparison results identifier that uniquely identifies an object instance associated with the data integrity record,

a field for a first object identifier that identifies a data object in the first data management system that corresponds to the object instance associated with the data integrity record, and

a field for a second object identifier that identifies a data object in the second data management system that corresponds to the object instance associated with the data integrity record.

39. The method of claim 38 wherein each data integrity record further includes:

a first indicator field that indicates whether the object instance was found in the first data management system, and

a second indicator field that indicates whether the object instance was found in the second data management system.

40. The method of claim 1 further comprising:

transforming identifiers of data objects identified from the second data management system to a format used by the first data management system, wherein

comparing identifiers of data objects identified from the first data management system with the identifiers of data objects identified from the second data management system comprises comparing identifiers of data objects identified from the first data management system with the transformed identifiers of data objects identified from the second data management system.

41. The method of claim 40 wherein:

the identifiers of data objects identified from the first data management system are global unique identifiers,

the identifiers of data objects identified from the second data management system are key identifiers, and

transforming identifiers of data objects identified from the second data management system to a format used by the first data management system comprises transforming key identifiers of data objects identified from the second data management system to global unique identifiers.

17. A computer system having embodied thereon a computer program configured to manage data integrity, the computer system comprising one or more code segments configured to:

identify data objects stored in a first data management system that meet a filter condition, each data object including an identifier that uniquely identifies a data object from other data objects in the first data management system, a set of attributes, and attribute values that correspond to attributes in the attribute set, and the filter condition comprising at least one value of an attribute, the attribute corresponding to an attribute in the attribute set;

identify data objects stored in a second data management system that meet the filter condition, each data object including an identifier that uniquely identifies a data object from other data objects in the second data management system, a set of attributes, and attribute values that correspond to attributes in the attribute set;

access data that identifies excluded data objects that are excluded from managing data integrity even when the excluded data objects meet the filter condition;

based on the accessed data, determine a subset of the data objects identified from the first data management system as meeting the filter condition that excludes the excluded data objects and a subset of the data objects identified from the second data management system as meeting the filter condition that excludes the excluded data objects;

compare identifiers from the subset of data objects identified from the first data management system with identifiers from the subset of data objects identified from the second data management system to determine whether each data object in one of the data management systems has a corresponding data object in the other data management system;

store, in electronic storage, comparison results information indicating results of the comparison of the identifiers from the subset of data objects identified from the first data management system with the identifiers from the subset of data objects identified from the second data management system, the comparison results information indicating whether each data object that is included in at least one of the subsets and meets the filter condition is stored in the first data management system, the second data management system, or both the first data management system and the second data management system;

access the comparison results information; and

manage the integrity of the identified data objects based on the accessed comparison results information such that each data object that is included in at least one of the subsets and meets the filter condition is included both in the first data management system and in the second data management system, managing the integrity comprising;

when a data object that is included in at least one of the subsets and meets the filter condition occurs only in the first data management system, sending the data object from the first data management system to the second data management system,

when a data object that is included in at least one of the subsets and meets the filter condition occurs only in the second data management system, sending the data object from the second data management system to the first data management system,

when a data object that is included in at least one of the subsets and meets the filter condition occurs in both the first data management system and in the second data management system, determining whether a first set of attribute values associated with the data object in the first data management system is equal to a second set of attribute values associated with the data object in the second data management system, and

when the first set of attribute values is not equal to the second set of attribute values, sending the data object from the first data management system to the second data management system.

18. The computer system of claim 17 wherein the one or more code segments configured to manage the integrity comprise one or more code segments configured to delete the data object from the second data management system and send the data object from the first data management system to the second data management system when the first set of attribute values is not equal to the second set of attribute values.

20. The computer system of claim 17 wherein the filter condition comprises one or more values for one or more attributes included in a data object.

21. The computer system of claim 17 wherein the filter condition comprises a data object identifier.

22. The computer system of claim 17 further comprising one or more code segments configured to:

extract, from the first data management system, only identifiers for a first group of data objects that include multiple attribute values, are included in the subset of the data objects identified from the first data management system, and meet the filter condition, extracting only identifiers comprising extracting identifiers for the first group of data objects without extracting other of the multiple attribute values,

extract, from the second data management system, only identifiers for a second group of data objects that include multiple attribute values, are included in the subset of the data objects identified from the second data management system, and meet the filter condition, extracting only identifiers comprising extracting identifiers for the second group of data objects without extracting other of the multiple attribute values,

wherein the one or more code segments configured to compare identifiers from the subset of data objects identified from the first data management system with identifiers from the subset of data objects identified from the second data management system to determine “

whether each data object in one of the data management systems has a corresponding data object in the other data management system comprise one or more code segments configured to compare only the extracted identifiers for the first group of data objects that meet the filter condition with the extracted identifiers for the second group of data objects that meet the filter condition.

23. The computer system of claim 17 further comprising one or more code segments configured to:

provide a user with an indication of the results of the comparison of identifiers;

permit the user to determine whether to proceed with managing the integrity of the identified data objects based on the indication of the results of the comparison of the identifiers; and

only after receiving a first indication from the user to proceed;

extract, from the first data management system, the other of the multiple attribute values from the first group of data objects that meet the filter condition,

extract, from the second data management system, the other of the multiple attribute values from the second group of data objects that meet the filter condition,

for data objects included in both the first group of data objects and the second group of data objects, compare the extracted attribute values from the first data management system with the extracted attribute values from the second data management system;

provide the user with an indication of the results of the comparison of the extracted attribute values;

permit the user to determine whether to proceed with managing the integrity of the identified data objects based on the indication of the results of the comparison of the extracted attribute values; and

only after receiving a second indication from the user to proceed, manage the data integrity of the identified data objects.

24. The computer system of claim 17 wherein a data object comprises an object instance, and the identifier is an object identifier.

25. The computer system of claim 17 wherein a data object comprises a data record, and the identifier is a record identifier.

26. The computer system of claim 17 wherein a data object comprises a row in a database table, and the identifier is a row identifier.

27. The computer system of claim 17 wherein a data object comprises a data segment, and the identifier is a data segment identifier.

28. The computer system of claim 17 further comprising one or more code segments configured to:

determine the number of data objects that occur in both the first data management system and the second data management system and meet the filter condition;

determine the number of data objects that occur in only the first data management system and meet the filter condition; and

determine the number of data objects that occur in only the second data management system and meet the filter condition.

29. The computer system of claim 28 further comprising one or more code segments configured to:

present the number of data objects that occur in both the first data management system and the second data management system and meet the filter condition;

present the number of data objects that occur in only the first data management system and meet the filter condition; and

present the number of data objects that occur in only the second data management system and meet the filter condition.

30. The computer system of claim 17 further comprising one or more code segments configured to:

extract from the first data management system a first group of data objects that meet the filter condition; and

extract from the second data management system a second group of data objects that that meet the filter condition.

31. The computer system of claim 17 further comprising one or more code segments configured to:

receive from the user an indication of one or more data objects that are associated both with the first group of data objects and with the second group of data objects, each single data object having a first set of attribute values associated with the first group of data objects and a second set of attribute values associated with the second group of data objects; and

provide the user with an indication of whether the first set of attribute values is equal to the second set of attribute values.

32. The computer system of claim 17 wherein the one or more code segments configured to manage the integrity of the data objects further comprise one or more code segments configured to transform one or more values for a set of attributes associated with a data object such that the set of attributes associated with the data object in the first data management system may be compared with the set of attributes associated with the data object in the second data management system.

19. The computer system of 17 wherein the filter condition is a user-definable filter condition.

33. A computer system having embodied thereon a computer program configured to manage data integrity, the computer system comprising one or more code segments configured to:

receive a filter condition from a user, the filter condition comprising at least one value of an attribute, the filter attribute corresponding to an attribute included in a set of attributes in one or more data objects stored in a leading data management system, each data object including an identifier that uniquely identifies a data object from other data objects in the leading data management system, and each data object including attribute values that correspond to attributes in the attribute set;

extract from the leading data management system a first group of identifiers of data objects, each data object meeting the filter condition;

extract, from a contrast data management system that includes stored data objects having i) an identifier that uniquely identifies a data object from other data objects in the contrast data management system, ii) a set of attributes and iii) attribute values that correspond to attributes in the attribute set, a second group of identifiers of data objects, each data object meeting the filter condition;

access data that identifies excluded data objects that are excluded from managing data integrity even when the excluded data objects meet the filter condition;

based on the accessed data, determine the data objects identified from the leading data management system as meeting the filter condition that excludes the excluded data objects and a subset of the data objects identified from the contrast data management system as meeting the filter condition that excludes the excluded data objects;

compare the first group of identifiers of data objects extracted from the leading data management system with the second group of identifiers of data objects extracted from the contrast data management system to determine whether each identifier in the first group represents the same data object as identified by an identifier in the second group of identifiers;

store, in electronic storage, comparison results information indicating results of the comparison of the first group of identifiers of data objects extracted from the leading data management system with the second group of identifiers of data objects extracted from the contrast data management system, the comparison results information indicating whether each data object that meets the filter condition is stored in the leading data management system, the contrast data management system, or both the leading data management system and the contrast data management system;

access the comparison results information;

based on the accessed comparison results information;

present a results table, each row in the results table representing a particular data object that meets the filter condition and including information indicative of whether the particular data object that meets the filter condition is stored in only the leading data management system, only the contrast data management system, or both the leading data management system and the contrast data management system;

determine a number of the data objects that meet the filter condition and are stored in only the leading data management system;

determine a number of the data objects that meet the filter condition and are stored in only the contrast data management system;

determine a number of the data objects that meet the filter condition and are stored in both the leading data management system and the contrast data management system;

display, along with the results table, the number of the data objects that meet the filter condition and are stored in only the leading data management system, the number of the data objects that meet the filter condition and are stored in only the contrast data management system, and the number of the data objects that meet the filter condition and are stored in both the leading data management system and the contrast data management system;

permit the user to determine whether to proceed with managing data integrity based on the presented results table, the displayed number of the data objects that meet the filter condition and are stored in only the leading data management system, the displayed number of the data objects that meet the filter condition and are stored in only the contrast data management system, and the displayed number of the data objects that meet the filter condition and are stored in both the leading data management system and the contrast data management system;

after receiving a first indication from the user to proceed, i) extract from the leading data management system a first group of attribute value sets, each attribute value set being associated with a single data object that is included in both the leading data management system and the contrast data management system and that meets the filter condition, ii) extract from the contrast data management system a second group of attribute value sets, each attribute value set being associated with a particular object that is included in both the leading data management system and the contrast data management system and that meets the filter condition, and iii) for each attribute value set extracted from the leading data management system, identify the attribute value set from the contrast data management system that represents the same data object and compare the attribute value set from the leading data management system with the attribute value set from the contrast data management system to determine whether the attribute value sets match,

update the results table to present, for each row in the results table representing a data object stored in both the leading data management system and the contrast data management an indication whether the attribute value set from the leading data management system matches the attribute value set from the contrast data management system;

determine a number of data objects that are stored in both the leading data management system and the contrast data management system and that have matching attribute value sets;

determine a number of data objects that are stored in both the leading data management system and the contrast data management system and that have differing attribute value sets;

display, along with the updated results table, the number of data objects that are stored in both the leading data management system and the contrast data management system and that have matching attribute value sets and the number of data objects that are stored in both the leading data management system and the contrast data management system and that have differing attribute value sets;

permit the user to determine whether to correct inconsistencies in data objects that meet the filter condition based on the updated results table, the displayed number of data objects that are stored in both the leading data management system and the contrast data management system and that have matching attribute value sets, and the displayed number of data objects that are stored in both the leading data management system and the contrast data management system and that have differing attribute value sets; and

after receiving a second indication from the user to proceed, correcting inconsistencies in data objects that meet the filter condition.

34. The computer system of claim 33 further comprising one or more code segments configured to receive an indication from a user to compare data objects in the leading data management system with data objects in the contrast data management system.

35. The computer system of claim 33 further comprising one or more code segments configured to:

determine which identifiers are associated with both the leading data management system and the contrast data management system and meet the filter condition;

determine which identifiers are associated with only the leading data management system and meet the filter condition; and

determine which identifiers are associated with only the contrast data management system and meet the filter condition.