Verifying data consistency

• US 10,176,216 B2
• Filed: 02/01/2016
• Issued: 01/08/2019
• Est. Priority Date: 02/01/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bourbonnais, Serge, Fan, Si Bin, Li, Xiao, Wierenga, Jonathan W., Zeng, Mai
• Primary Examiner(s): Corrielus, Jean M
• Application Number:
  US15/012,103
• Publication Number:
  US 20170220625A1
• Time in Patent Office: 1,072 Days
• Field of Search: 707690
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/1805 Append-only file systems, e.g. using logs or journals to store data
• CPC Class Codes: G06F 16/2255 Hash tables
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/2315 Optimistic concurrency control
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting

Abstract

A method for verifying data consistency between update-in-place data structures and append-only data structures containing change histories associated with the update-in-place data structures is provided. The method includes loading data from an update-in-place data structure to a first set of hash buckets in a processing platform, loading data from append-only data structures to a second set of hash buckets in the processing platform, performing a bucket-level comparison between the data in the first set of hash buckets and the data in the second set of has buckets, and generating a report based on the bucket-level comparison.

Claims

1. A method for verifying data consistency between update-in-place data structures and append-only data structures containing change histories associated with the update-in-place data structures, the method comprising:

loading data from a first update-in-place data structure to a first set of hash buckets in a processing platform, wherein the data from the first update-in-place data structure comprises a first set of key values that corresponds to rows of data in the first update-in-place data structure, and wherein loading the data from the first update-in-place data structure to the first set of hash buckets is based on a first set of hash values associated with the first set of key values;

loading data from the append-only data structures to a second set of hash buckets in the processing platform;

performing a bucket-level comparison between the data in the first set of hash buckets and the data in the second set of hash buckets;

generating an intermediate report based on the bucket-level comparison, wherein generating the intermediate report based on the bucket-level comparison comprises;

determining an update occurred to the first update-in-place data structure during the bucket-level comparison;

identifying transient differences between the first update-in-place data structure and the append-only data structures, wherein the transient differences comprise differences caused by either in-flight transactions, by rollback transactions, or by in-flight transactions and by rollback transactions committed at the first update-in-place data structure after loading the data from the first update-in-place data structure to the first set of hash buckets in the processing platform; and

removing the transient differences from the intermediate report listing differences between the first update-in-place data structure and the append-only data structures; and

generating a final report based on the intermediate report and removal of the identified transient differences, wherein the final report comprises persistent differences between the first update-in-place data structure and the append-only data structures and omits the identified transient differences removed from the intermediate report, wherein the final report is generated for live comparison of the first update-in-place data structure and the append-only data structures, and wherein the differences are inserted into a second update-in-place data structure that is associated with the first update-in-place data structure.

2. The method according to claim 1, wherein the data from the append-only data structures comprises a second set of key values that corresponds to rows of data in the append-only data structures, and wherein loading the data from the append-only data structures to the second set of hash buckets is based on a second set of hash values associated with the second set of key values.

3. The method according to claim 2, wherein the first set of hash values and the second set of hash values are determined based on a common hash function.

4. The method according to claim 3, wherein the bucket-level comparison is performed on buckets from the first set of hash buckets sharing common hash values with buckets from the second set of hash buckets.

5. The method according to claim 4, wherein the data from the first update-in-place data structure further comprises a first set of checksum values that corresponds to rows of data in the first update-in-place data structure, and wherein the data from the append-only data structures further comprises a second set of checksum values that corresponds to rows of data in the append-only data structures.

6. The method according to claim 5, wherein the intermediate report based on the bucket-level comparison comprises:

a third set of key values corresponding to differences between the first update-in-place data structure and the append-only data structures;

a set of difference types corresponding to the third set of key values;

checksum values from the first set of checksum values corresponding to the third set of key values; and

checksum values from the second set of checksum values corresponding to the third set of key values.

7. The method according to claim 4, further comprising:

generating a first set of checksum values that corresponds to rows of data in the first update-in-place data structure; and

generating a second set of checksum values that corresponds to rows of data in the append-only data structures, wherein generating the first set of checksum values and generating the second set of checksum values are based on a common function.

8. The method according to claim 1, wherein loading the data from the first update-in-place data structure to the first set of hash buckets and loading the data from the append-only data structures to the second set of hash buckets is performed in parallel.

9. The method according to claim 1, wherein the bucket-level comparison is performed in parallel.

10. The method according to claim 1, wherein the first update-in-place data structure is a relational database management systems (RDBMS).

11. A computer program product for verifying data consistency between update-in-place data structures and append-only data structures containing change histories associated with the update-in-place data structures, the computer program product comprising at least one computer readable non-transitory storage medium having computer readable program instructions thereon for execution by a processor, the computer readable program instructions comprising program instructions for:

loading data from a first update-in-place data structure to a first set of hash buckets in a processing platform, wherein the data from the first update-in-place data structure comprises a first set of key values that corresponds to rows of data in the first update-in-place data structure, and wherein loading the data from the first update-in-place data structure to the first set of hash buckets is based on a first set of hash values associated with the first set of key values;

loading data from the append-only data structures to a second set of hash buckets in the processing platform;

performing a bucket-level comparison between the data in the first set of hash buckets and the data in the second set of hash buckets;

generating an intermediate report based on the bucket-level comparison, wherein generating the intermediate report based on the bucket-level comparison comprises;

determining an update occurred to the first update-in-place data structure during the bucket-level comparison;

identifying transient differences between the first update-in-place data structure and the append-only data structures, wherein the transient differences comprise differences caused by either in-flight transactions, by rollback transactions, or by in-flight transactions and by rollback transactions committed at the first update-in-place data structure after loading the data from the first update-in-place data structure to the first set of hash buckets in the processing platform; and

removing the transient differences from the intermediate report listing differences between the first update-in-place data structure and the append-only data structures; and

generating a final report based on the intermediate report and removal of the identified transient differences, wherein the final report comprises persistent differences between the first update-in-place data structure and the append-only data structures and omits the identified transient differences removed from the intermediate report, wherein the final report is generated for live comparison of the first update-in-place data structure and the append-only data structures, and wherein the differences are inserted into a second update-in-place data structure that is associated with the first update-in-place data structure.

12. The computer program product according to claim 11, wherein the data from the append-only data structures comprises a second set of key values that corresponds to rows of data in the append-only data structures, and wherein loading the data from the append-only data structures to the second set of hash buckets is based on a second set of hash values associated with the second set of key values.

13. A computer system for verifying data consistency between update-in-place data structures and append-only data structures containing change histories associated with the update-in-place data structures, the computer system comprising:

at least one processing unit;

at least one computer readable memory;

at least one computer readable tangible, non-transitory storage medium; and

program instructions stored on the at least one computer readable tangible, non-transitory storage medium for execution by the at least one processing unit via the at least one computer readable memory, wherein the program instructions comprise program instructions for;

loading data from a first update-in-place data structure to a first set of hash buckets in a processing platform, wherein the data from the first update-in-place data structure comprises a first set of key values that corresponds to rows of data in the first update-in-place data structure, and wherein loading the data from the first update-in-place data structure to the first set of hash buckets is based on a first set of hash values associated with the first set of key values;

loading data from the append-only data structures to a second set of hash buckets in the processing platform;

performing a bucket-level comparison between the data in the first set of hash buckets and the data in the second set of hash buckets;

generating an intermediate report based on the bucket-level comparison, wherein generating the intermediate report based on the bucket-level comparison comprises;

determining an update occurred to the first update-in-place data structure during the bucket-level comparison;

identifying transient differences between the first update-in-place data structure and the append-only data structures, wherein the transient differences comprise differences caused by either in-flight transactions, by rollback transactions, or by in-flight transactions and by rollback transactions committed at the first update-in-place data structure after loading the data from the first update-in-place data structure to the first set of hash buckets in the processing platform; and

removing the transient differences from the intermediate report listing differences between the first update-in-place data structure and the append-only data structures;

and

generating a final report based on the intermediate report and removal of the identified transient differences, wherein the final report comprises persistent differences between the first update-in-place data structure and the append-only data structures and omits the identified transient differences removed from the intermediate report, wherein the final report is generated for live comparison of the first update-in-place data structure and the append-only data structures, and wherein the differences are inserted into a second update-in-place data structure that is associated with the first update-in-place data structure.

14. The computer system according to claim 13, wherein the data from the append-only data structures comprises a second set of key values that corresponds to rows of data in the append-only data structures, and wherein loading the data from the append-only data structures to the second set of hash buckets is based on a second set of hash values associated with the second set of key values.