Maintaining data consistency in mirrored cluster storage systems with write-back cache

• US 8,244,999 B1
• Filed: 03/14/2011
• Issued: 08/14/2012
• Est. Priority Date: 01/30/2007
• Status: Active Grant

• Current Assignee: AmZetta Technologies, LLC (American Megatrends International, LLC)
• Original Assignee: American Megatrends Incorporated
• Inventors: Chatterjee, Paresh, Balakrishnan, Narayanan, Jayaraman, Raja
• Primary Examiner(s): DUDEK JR, EDWARD J
• Application Number:
  US13/046,857
• Time in Patent Office: 519 Days
• Field of Search: 711/162, 711/135, 711/E12.103, 707/655
• US Class Current: 711/162
• CPC Class Codes: G06F 11/2064 while ensuring consistency
• CPC Class Codes: G06F 11/2082 Data synchronisation

Abstract

Techniques for maintaining mirrored storage cluster data consistency on systems with write back cache can employ write-intent logging and periodic log clearing. The techniques can track any outstanding I/Os, data in caches, and data that has gone out of sync between mirrored nodes due to link failures. The techniques can ensure that a power failure on any of the storage nodes does not result in inconsistent data among the storage nodes. The techniques can use gate bitmaps to log outstanding I/Os. The gate bitmaps may be periodically cleared. The clearing period may be selected as to allow enough time for all caches to be flushed and for all synchronizing I/Os to complete. Efficient queuing of I/O requests in conjunction with the efficient operation of multiple gate bitmaps can reduce performance impacts of the gating or logging operations.

Claims

1. A method comprising:

loading a gate bitmap into memory, wherein the gate bitmap comprises a plurality of bits and each bit corresponds to a storage area of a plurality of storage areas within a primary networked storage node;

logging, on the primary networked storage node, an intent to write data to a storage area of the primary networked storage node by setting a bit in the gate bitmap corresponding to the storage area;

writing the data to the primary networked storage node after logging the intent to write data;

relaying a request to a secondary networked storage node to mirror the data after logging the intent to write data; and

after a delay period completes, clearing the bit set in the gate bitmap to protect an underlying write-back cache,

wherein the gate bitmap has an associated bit vector, and the bit vector has one or more bits that indicate a size of the storage areas corresponding to each bit of the gate bitmap.

2. A method comprising:

loading a gate bitmap into memory, wherein the gate bitmap comprises a plurality of bits and each bit corresponds to a storage area of a plurality of storage areas within a primary networked storage node;

logging, on the primary networked storage node, an intent to write data to a storage area of the primary networked storage node by setting a bit in the gate bitmap corresponding to the storage area;

writing the data to the primary networked storage node after logging the intent to write data;

relaying a request to a secondary networked storage node to mirror the data after logging the intent to write data; and

after a delay period completes, clearing the bit set in the gate bitmap to protect an underlying write-back cache,

wherein the gate bitmap has an associated bit vector, and the bit vector has one or more bits that indicate a sequence number of the gate bitmap.

3. The method of claim 2, wherein logging further comprises writing the gate bitmap to a disk, and updating the sequence number indicated by the one or more bits of the bit vector associated with the gate bitmap.

4. The method of claim 3, wherein the gate bitmap is alternatively written to two disk locations on the disk.

5. The method of claim 4, wherein loading a gate bitmap into memory comprises:

examining the sequence numbers indicated by the one or more bits of the bit vectors associated with the gate bitmaps written to each of the two disk locations; and

loading the gate bitmap into memory with the associated bit vector whose one or more bits indicate a greatest sequence number.

6. The method of claim 2, wherein the gate bitmap is one of a plurality of gate bitmaps and each gate bitmap has an associated sequence number, and loading a gate bitmap into memory comprises loading the gate bitmap into memory with a greatest associated sequence number.

7. A method comprising:

loading a gate bitmap into memory, wherein the gate bitmap comprises a plurality of bits and each bit corresponds to a storage area of a plurality of storage areas within a primary networked storage node;

logging, on the primary networked storage node, an intent to write data to a storage area of the primary networked storage node by setting a bit in the gate bitmap corresponding to the storage area;

writing the data to the primary networked storage node after logging the intent to write data;

relaying a request to a secondary networked storage node to mirror the data after logging the intent to write data; and

after a delay period completes, clearing the bit set in the gate bitmap to protect an underlying write-back cache,

wherein the secondary storage node comprises a plurality of secondary storage nodes, the gate bitmap comprises a plurality of gate bitmaps, and each gate bitmap of the plurality of gate bitmaps corresponds to a secondary storage node of the plurality of secondary storage nodes.

8. A computer storage medium having computer-executable instructions stored thereon which, when executed by a computer system, cause the computer system to:

load a gate bitmap, wherein the gate bitmap comprises a plurality of bits and each bit corresponds to a storage area from a plurality of storage areas of the computer system;

receive an input/output (I/O) request, wherein the I/O request has a related storage area from the plurality of storage areas;

record the I/O request in the bit of the gate bitmap corresponding to the storage area that is related to the I/O request;

store the gate bitmap to a disk;

execute the I/O request after the gate bitmap has been stored to the disk;

relay the I/O request to a mirrored computer system for execution;

wait for a delay period to complete; and

clear the gate bitmap after the wait,

wherein the gate bitmap has an associated bit vector, and the bit vector has one or more bits that indicate a size of the storage areas corresponding to each bit of the gate bitmap.

9. A computer storage medium having computer-executable instructions stored thereon which, when executed by a computer system, cause the computer system to:

load a gate bitmap, wherein the gate bitmap comprises a plurality of bits and each bit corresponds to a storage area from a plurality of storage areas of the computer system;

receive an input/output (I/O) request, wherein the I/O request has a related storage area from the plurality of storage areas;

record the I/O request in the bit of the gate bitmap corresponding to the storage area that is related to the I/O request;

store the gate bitmap to a disk;

execute the I/O request after the gate bitmap has been stored to the disk;

relay the I/O request to a mirrored computer system for execution;

wait for a delay period to complete; and

clear the gate bitmap after the wait,

wherein the gate bitmap has an associated bit vector, and the bit vector has one or more bits that indicate a sequence number of the gate bitmap.

10. The computer storage medium of claim 9, wherein storing the gate bitmap to the disk further comprises updating the sequence number indicated by the one or more bits of the bit vector associated with the gate bitmap.

11. The computer storage medium of claim 10, wherein storing the gate bitmap to the disk comprises alternatively storing the gate bitmap at two disk locations on the disk.

12. The computer storage medium of claim 11, wherein loading a gate bitmap into memory comprises:

examining the sequence numbers indicated by the one or more bits of the bit vectors associated with the gate bitmaps stored at each of the two disk locations; and

loading the gate bitmap into memory with the associated bit vector whose one or more bits indicate a greatest sequence number.

13. The computer storage medium of claim 9, wherein the gate bitmap is one of a plurality of gate bitmaps and each gate bitmap has an associated sequence number, and loading a gate bitmap into memory comprises loading the gate bitmap into memory with a greatest associated sequence number.

14. A system comprising:

a disk;

a secondary storage node; and

a primary storage node adapted to;

load a gate bitmap into memory from the disk, wherein the gate bitmap comprises a plurality of bits and each bit corresponds to a storage area of a plurality of storage areas within the primary storage node;

log an intent to write data to a storage area of the primary storage node by setting a bit in the gate bitmap corresponding to the storage area;

write the data after logging the intent to write data;

relay a request to the secondary storage node to mirror the data after logging the intent to write data; and

after a delay period completes, clearing the bit set in the gate bitmap to protect an underlying write-back cache,

wherein the gate bitmap has an associated bit vector, and the bit vector has one or more bits that indicate a size of the storage areas corresponding to each bit of the gate bitmap.

15. A system comprising:

a disk;

a secondary storage node; and

a primary storage node adapted to;

load a gate bitmap into memory from the disk, wherein the gate bitmap comprises a plurality of bits and each bit corresponds to a storage area of a plurality of storage areas within the primary storage node;

log an intent to write data to a storage area of the primary storage node by setting a bit in the gate bitmap corresponding to the storage area;

write the data after logging the intent to write data;

relay a request to the secondary storage node to mirror the data after logging the intent to write data; and

after a delay period completes, clearing the bit set in the gate bitmap to protect an underlying write-back cache,

wherein logging further comprises alternatively writing the gate bitmap to two locations on the disk.

16. A system comprising:

a disk;

a secondary storage node; and

a primary storage node adapted to;

load a gate bitmap into memory from the disk, wherein the gate bitmap comprises a plurality of bits and each bit corresponds to a storage area of a plurality of storage areas within the primary storage node;

log an intent to write data to a storage area of the primary storage node by setting a bit in the gate bitmap corresponding to the storage area;

write the data after logging the intent to write data;

relay a request to the secondary storage node to mirror the data after logging the intent to write data; and

after a delay period completes, clearing the bit set in the gate bitmap to protect an underlying write-back cache,

wherein the gate bitmap is one of a plurality of gate bitmaps and each gate bitmap has an associated sequence number, and loading a gate bitmap into memory comprises loading the gate bitmap into memory with a greatest associated sequence number.

17. A system comprising:

a disk;

a secondary storage node; and

a primary storage node adapted to;

load a gate bitmap into memory from the disk, wherein the gate bitmap comprises a plurality of bits and each bit corresponds to a storage area of a plurality of storage areas within the primary storage node;

log an intent to write data to a storage area of the primary storage node by setting a bit in the gate bitmap corresponding to the storage area;

write the data after logging the intent to write data;

relay a request to the secondary storage node to mirror the data after logging the intent to write data; and

after a delay period completes, clearing the bit set in the gate bitmap to protect an underlying write-back cache,

wherein the secondary storage node comprises a plurality of secondary storage nodes, the gate bitmap comprises a plurality of gate bitmaps, and each gate bitmap of the plurality of gate bitmaps corresponds to a secondary storage node of the plurality of secondary storage nodes.