Maintaining consistency among copies of a logical storage volume in a distributed storage system

• US 10,534,549 B2
• Filed: 09/19/2017
• Issued: 01/14/2020
• Est. Priority Date: 09/19/2017
• Status: Active Grant

• Current Assignee: Rakuten Symphony Incorporated (Rakuten Group, Inc.)
• Original Assignee: Robin Systems, Inc. (Rakuten Group, Inc.)
• Inventors: Singh, Gurmeet, Patel, Ripulkumar Hemantbhai, Seetala, Partha Sarathi
• Primary Examiner(s): Matin, Tasnima
• Application Number:
  US15/709,296
• Publication Number:
  US 20190087105A1
• Time in Patent Office: 847 Days
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/1004 to protect a block of data words, e.g. CRC or checksum G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A system maintains a consistency database that maintains a status (current, down, stale) for copies of logical storage volumes stored on storage nodes. As failures are detected, the consistency database is updated. Copies are synchronized with one another using information in the consistency database. Write operations on a primary node for a slice of a logical storage node are assigned a virtual block address (VBA) that is mapped to a logical block address (LBA) within the slice. Consistency of the VBAs of the primary node and that of a secondary node is evaluated and used to detect currency. VBA holes are detected and corresponding write commands resent to maintain currency. Physical segments on the primary node are assigned virtual segment identifiers (VSID) that are maintained consistent with VSIDs on clone nodes so that they can be used for garbage collection and synchronization.

Claims

1. A system comprising one or more processing devices and one or more memory devices operably coupled to the one or more processing devices, the one or more memory devices storing executable code effective to cause the one or more processing devices to:

receive first write input/output operations (IOPs) from one or more compute nodes; and

for each first write input/output operation (IOP) from one of the one or more compute nodes, execute the each first write IOP with respect to a first copy of a first storage unit referenced by the each first write IOP by—

assigning a first virtual block address (VBA) to a logical block address (LBA) referenced in the each first write IOP according to a VBA counter;

incrementing the VBA counter;

storing an association between the LBA and the first VBA;

writing data from the each first write IOP to a first physical storage location;

storing an association between the first physical storage location and the first VBA; and

transmitting the each first write IOP to a second storage node of a plurality of storage nodes with the first VBA;

receive second write IOPs from the plurality of storage nodes;

for each second write IOP received from a first storage node of the plurality of storage nodes, execute the each second write IOP with respect to a second copy of a second storage unit referenced by the each second write IOP by—

comparing a second VBA included with the each second write IOP to a second VBA counter;

if the second VBA is not consistent with the second VBA counter, notifying the first storage node that the second copy is not current;

if the second VBA is consistent with the second VBA counter—

incrementing the second VBA counter;

storing an association between a second LBA in the each second write IOP and the second VBA;

writing data from the each second write IOP to a second physical storage location;

storing an association between the second physical storage location and the VBA; and

transmitting an acknowledgment to the first storage node.

2. The system of claim 1, wherein the executable code is further effective to cause the one or more processing devices to, for each second write IOP received from the first storage node, execute the each second write IOP by:

if the second VBA is not consistent with the second VBA counter, refraining from executing the each second write IOP with respect to the second copy of the second storage unit referenced by the each second write IOP.

3. The system of claim 2, wherein the executable code is further effective to cause the one or more processing devices to, for each second write IOP received from the first storage node, execute the each second write IOP by, if the second VBA is not consistent with the VBA counter:

identifying a range of VBAs according to a difference between the second VBA and the VBA counter;

receiving third write IOPs corresponding to VBAs from the range of VBAs from the first storage node; and

executing each of the third write IOP and incrementing the VBA counter for each of the third write IOPs.

4. The system of claim 1, wherein the executable code is further effective to cause the one or more processing devices to write the data from the each first write IOP to the first physical location by:

if a first current segment allocated to a storage unit referenced in the each first write IOP is full, the first current segment having a current VSID (virtual segment identifier) associated with it in a first segment map—

allocating a first new segment of physical memory to the storage unit referenced in the each first write IOP as the first current segment;

assigning a new VSID (virtual segment identifier) from a VSID counter as the current VSID segment; and

incrementing the VSID counter;

storing a first association between the storage unit, the first new segment, and the new VSID in the first segment map; and

writing the data from the each first write TOP to the first current segment.

5. The system of claim 4, wherein the executable code is further effective to cause the one or more processing devices to transmit the current VSID with the each first write TOP to the second storage node of the plurality of storage nodes.

6. The system of claim 5, wherein the second storage node of the plurality of storage nodes is programmed to, for each first write TOP and current VSID received—

if the current VSID is not allocated to the storage unit referenced in the each first write TOP in a second segment map of the second storage node, the storage unit having a second current segment of physical memory allocated on the second storage node—

allocate a second new segment of physical memory to the storage unit referenced in the each first write TOP segment as the second current segment; and

store a second association between the storage unit, the second new segment, and the new VSID in the second segment map; and

write the data from the each first write TOP to the second current segment.

7. The system of claim 6, wherein the executable code is further effective to cause the one or more processing devices to, if an amount of data in the first new segment is below a threshold:

include, in the first write IOPs, copying write IOPs referencing valid data from the first new segment;

process the copying write IOPs with the first write IOPs in a same manner as the first write IOPs; and

transmit an instruction to the second storage node to free the new VSID.

8. The system of claim 6, wherein the executable code is further effective to cause the one or more processing devices to, if an amount of data written to the first new segment has filled the first new segment:

compute a first checksum of the first new segment; and

transmit the first checksum to the second storage node;

wherein the second storage node is further programmed to;

compute a second checksum of the second new segment; and

if the second checksum is not identical to the first checksum, invoke copying of the first new segment to the second new segment.

9. The system of claim 1, wherein the executable code is further effective to cause the one or more processing devices to:

for each first write IOP from one of the one or more compute nodes, acknowledge completion of the each first write IOP to the one of the one or more compute nodes only after acknowledging completion of all preceding first write IOPS of the first write IOPs regardless of an order of completion of the first write IOPs.

10. A method comprising:

receiving, by a subject storage node comprising a processing device, first write input/output operations (IOPs) from one or more compute nodes;

for each first write input/output operation (TOP) from one of the one or more compute nodes, executing, by the subject storage node, the each first write IOP with respect to a first copy of a first storage unit referenced by the each first write IOP by—

assigning a first virtual block address (VBA) to a logical block address (LBA) referenced in the each first write IOP according to a VBA counter;

incrementing the VBA counter;

storing an association between the LBA and the first VBA;

writing data from the each first write IOP to a first physical storage location;

storing an association between the first physical storage location and the first VBA;

transmitting the each first write IOP to a second storage node of a plurality of storage nodes with the first VBA;

receiving, by the subject storage node, second write IOPs from the plurality of storage nodes;

for each second write IOP received from a first storage node of the plurality of storage nodes, executing, by the subject storage node, the each second write IOP with respect to a second copy of a second storage unit referenced by the each second write IOP by—

comparing a second VBA included with the each second write IOP to a second VBA counter;

when the second VBA is not consistent with the second VBA counter, notifying the first storage node that the second copy is not current;

when the second VBA is consistent with the second VBA counter—

incrementing the second VBA counter;

storing an association between a second LBA in the each second write TOP and the second VBA;

writing data from the each second write TOP to a second physical storage location;

storing an association between the second physical storage location and the VBA; and

transmitting an acknowledgment to the first storage node.

11. The method of claim 10, wherein the subject storage node is further programmed to:

when the second VBA is not consistent with the second VBA counter, refraining from executing the each second write TOP with respect to the second copy of the second storage unit referenced by the each second write TOP.

12. The method of claim 11, further comprising, when the second VBA is not consistent with the VBA counter:

identifying, by the subject storage node, a range of VBAs according to a difference between the second VBA and the VBA counter;

receiving, by the subject storage node, third write IOPs corresponding to VBAs from the range of VBAs from the first storage node; and

executing, by the subject storage node, each of the third write IOP and incrementing the VBA counter for each of the third write IOPs.

13. The method of claim 10, further comprising writing, by the subject storage node, the data from the each first write IOP to the first physical location by:

when a first current segment allocated to a storage unit referenced in the each first write IOP is full, the first current segment having a current VSID (virtual segment identifier) associated with it in a first segment map—

allocating a first new segment of physical memory to the storage unit referenced in the each first write IOP as the first current segment;

assigning a new VSID (virtual segment identifier) from a VSID counter as the current VSID segment; and

incrementing the VSID counter;

storing a first association between the storage unit, the first new segment, and the new VSID in the first segment map; and

writing the data from the each first write IOP to the first current segment.

14. The method of claim 13, further comprising transmitting, by the subject storage node, the current VSID with the each first write IOP to the second storage node of the plurality of storage nodes.

15. The method of claim 14, further comprising, by the subject storage node, for each first write IOP and current VSID received—

when the current VSID is not allocated to the storage unit referenced in the each first write IOP in a second segment map of the second storage node, the storage unit having a second current segment of physical memory allocated on the second storage node—

allocating a second new segment of physical memory to the storage unit referenced in the each first write IOP segment as the second current segment; and

storing a second association between the storage unit, the second new segment, and the new VSID in the second segment map; and

writing the data from the each first write IOP to the second current segment.

16. The method of claim 15, further comprising, by the subject storage node, when an amount of data in the first new segment is below a threshold:

including, in the first write IOPs, copying write IOPs referencing valid data from the first new segment;

processing the copying write IOPs with the first write IOPs in a same manner as the first write IOPs; and

transmitting an instruction to the second storage node to free the new VSID.

17. The method of claim 15, further comprising, by the subject storage node, when an amount of data written to the first new segment has filled the first new segment:

computing a first checksum of the first new segment; and

transmitting the first checksum to the second storage node; and

by the second storage node;

computing a second checksum of the second new segment; and

when the second checksum is not identical to the first checksum, invoking copying of the first new segment to the second new segment.

18. The method of claim 10, further comprising, by the subject storage node:

for each first write IOP from one of the one or more compute nodes, acknowledging completion of the each first write IOP to the one of the one or more compute nodes only after completion of all preceding first write IOPS of the first write IOPs referencing the first storage unit regardless of an order of completion of the first write IOPs.