VERIFYING DATA CONSISTENCY

• US 20170220628A1
• Filed: 12/06/2016
• Published: 08/03/2017
• Est. Priority Date: 02/01/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Li, Xiao, Wierenga, Jonathan W., Zeng, Mai, Fan, Si Bin, Bourbonnais, Serge
• Granted Patent: US 10,083,202 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/1805 Append-only file systems, e.g. using logs or journals to store data
• CPC Class Codes: G06F 16/2255 Hash tables
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/2315 Optimistic concurrency control
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting

Abstract

A method for verifying data consistency between update-in-place data structures and append-only data structures containing change histories associated with the update-in-place data structures is provided. The method includes loading data from an update-in-place data structure to a first set of hash buckets in a processing platform, loading data from append-only data structures to a second set of hash buckets in the processing platform, performing a bucket-level comparison between the data in the first set of hash buckets and the data in the second set of has buckets, and generating a report based on the bucket-level comparison.

Claims

1. A computer system for verifying data consistency between update-in-place data structures and append-only data structures containing change histories associated with the update-in-place data structures, comprising one or more computer devices each having one or more processors and one or more tangible storage devices;

and a program embodied on at least one of the one or more storage devices, the program having a plurality of program instructions for execution by the one or more processors, the program instructions comprising instructions for;

loading data from an update-in-place data structure to a first set of hash buckets in a processing platform;

loading data from append-only data structures to a second set of hash buckets in the processing platform;

performing a bucket-level comparison between the data in the first set of hash buckets and the data in the second set of has buckets;

generating an initial report based on the bucket level comparison;

determining an update occurred during the bucket level comparison;

removing from the initial report transient differences between the update-in-place data structure and the append-only data structures, wherein the transient differences comprise differences caused by rollback transactions committed at the update-in-place data structure after loading the data from the update-in-place data structure to the first set of hash buckets in the processing platform, and wherein the removing comprises a row-by-row re-fetch from the update-in-place data structure in an isolation level higher than a cursor stable isolation level; and

generating a final report based on the initial report and removal of the transient differences.