Storage scheme for a distributed storage system

• US 10,423,344 B2
• Filed: 09/19/2017
• Issued: 09/24/2019
• Est. Priority Date: 09/19/2017
• Status: Active Grant

• Current Assignee: Rakuten Symphony Incorporated (Rakuten Group, Inc.)
• Original Assignee: Robin Systems, Inc. (Rakuten Group, Inc.)
• Inventors: Singh, Gurmeet, Patel, Ripulkumar Hemantbhai, Seetala, Partha Sarathi
• Primary Examiner(s): Matin, Tasnima
• Application Number:
  US15/709,269
• Publication Number:
  US 20190087104A1
• Time in Patent Office: 735 Days
• CPC Class Codes: G06F 11/1004 to protect a block of data words, e.g. CRC or checksum G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A system maintains a consistency database that maintains a status (current, down, stale) for copies of logical storage volumes stored on storage nodes. As failures are detected, the consistency database is updated. Copies are synchronized with one another using information in the consistency database. Write operations on a primary node for a slice of a logical storage node are assigned a virtual block address (VBA) that is mapped to a logical block address (LBA) within the slice. Consistency of the VBAs of the primary node and that of a secondary node is evaluated and used to detect currency. VBA holes are detected and corresponding write commands resent to maintain currency. Physical segments on the primary node are assigned virtual segment identifiers (VSID) that are maintained consistent with VSIDs on clone nodes so that they can be used for garbage collection and synchronization.

Claims

1. A system comprising:

one or more compute nodes executing one or more applications;

a plurality of storage nodes each hosting one or more storage devices;

a consistency database manager hosting a consistency database, the consistency database storing, for each storage node of the plurality of storage nodes—

an operational status of the each storage node;

a listing of one or more storage units assigned to the each storage node; and

an update status for each storage one of the one or more storage units assigned to the each storage node;

wherein the consistency database manager is programmed to (a) receive notifications from each node of the one or more compute nodes and the plurality of storage nodes and (b) update the consistency database according to the notifications, each notification indicating at least one of;

that a source of the each notification is not current; and

that a storage node of the plurality of storage nodes is not responsive to the source of the each notification;

wherein each storage node of the plurality of storage nodes is further programmed to, for each first write IOP (input/output operation) from a first compute node of the one or more compute nodes, execute the each first write IOP with respect to a first copy of a first storage unit stored by the each storage node and referenced by the each first write IOP by—

assigning a first virtual block address (VBA) to a logical block address (LBA) referenced in the each first write IOP according to a first VBA counter;

incrementing the first VBA counter;

storing an association between the LBA and the first VBA;

writing data from the each first write IOP to a first physical storage location;

storing an association between the first physical storage location and the first VBA; and

transmitting the each first write IOP to a second storage node of the plurality of storage nodes with the first VBA.

2. The system of claim 1, wherein the consistency database manager is further programmed to, in response to failure of a first storage node of the plurality of storage nodes:

identify a current copy of a first storage unit of the one or more storage units assigned to the first storage node in the consistency database;

invoke storage of another copy of the current copy on a different storage node of the plurality of storage nodes; and

update an entry in the consistency database for the different storage node to include the first unit of storage in the listing of the one or more storage units assigned to the each storage node.

3. The system of claim 1, wherein the consistency database manager is further programmed to, in response to detecting from the consistency database that a first copy of a first storage unit stored by a first storage node of the plurality of storage nodes is not current:

identify a current copy of the first storage unit in the listing of a second storage node of the plurality of storage nodes;

invoke synchronization of the first copy with the current copy; and

update an entry in the consistency database for the first storage node to indicate that the first copy is current.

4. The system of claim 1, wherein the consistency database manager is further programmed to, in response to receiving an IOP (input/output operation) from a first compute node of the one or more compute nodes:

identify a storage unit referenced in the IOP;

identify a first storage node of the plurality of storage nodes that is listed in the consistency database as being a leader for that storage unit; and

transmit a reference to the first storage node to the first compute node.

5. The system of claim 1, wherein each storage node of the plurality of storage nodes is further programmed to:

receive a write IOP (input/output operation) from a compute node of the one or more compute nodes;

execute the write IOP with respect to a first copy of a first storage unit stored by the each storage node;

transmit the write IOP to a second storage node of the plurality of storage nodes; and

if no acknowledgment of the write IOP is received form the second storage node, transmit a first notification of the notifications indicating that the second storage node is at least one of (c) not operational and (d) not current with respect to the first storage unit.

6. The system of claim 1, wherein each storage node of the plurality of storage nodes is further programmed to, for each second write IOP received from a different storage node of the plurality of storage nodes, execute the each second write IOP with respect to a second copy of a second storage unit stored by the each storage node and referenced by the each second write IOP by—

comparing a second VBA included with the each second write IOP to a second VBA counter;

if the second VBA is not consistent with the second VBA counter, notifying the different storage node that the each storage node is not current;

if the second VBA is consistent with the second VBA counter—

incrementing the second VBA counter;

storing an association between a second LBA included in the each second write IOP and the second VBA;

writing data from the each second write IOP to a second physical storage location;

storing an association between the second physical storage location and the VBA; and

transmitting an acknowledgment to the different storage node.

7. The system of claim 6, wherein each storage node of the plurality of storage nodes is further programmed to, for each second write IOP received from the different storage node of the plurality of storage nodes, execute the each second write IOP by:

if the second VBA is not consistent with the second VBA counter, refraining from executing the each second write IOP with respect to the second copy of the second storage unit referenced by the each second write IOP.

8. The system of claim 7, wherein each storage node of the plurality of storage nodes is further programmed to, if the second VBA is not consistent with the VBA counter:

identify a range of VBAs according to a difference between the second VBA and the second VBA counter;

request and receive third write IOPs including VBAs from the range of VBAs from the different storage node; and

execute each third write IOP and incrementing the VBA counter for each third write IOP.

9. The system of claim 1, wherein each storage node of the plurality of storage nodes is further programmed to, for each first write IOP received from the first compute node of the one or more compute nodes, execute the each first write IOP with respect to a first copy of a first storage unit stored by the each storage node and referenced by the each first write IOP by—

following execution of the each first write IOP, acknowledging execution of the each first write IOP to the first compute node only after acknowledging execution of all previously received IOPs referencing the first storage unit.

10. A method comprising:

providing one or more compute nodes executing one or more applications;

providing a plurality of storage nodes each hosting one or more storage devices;

storing, by a computing device, a consistency database storing, for each storage node of the plurality of storage nodes—

an operational status of the each storage node;

a listing of one or more storage units assigned to the each storage node; and

an update status for each storage one of the one or more storage units assigned to the each storage node;

receiving, by the computing device, notifications from each node of the one or more compute nodes and the plurality of storage nodes;

updating, by the computing device, the consistency database according to the notifications, each notification indicating at least one of;

that a source of the each notification is not current; and

that a storage node of the plurality of storage nodes is not responsive to the source of the each notification; and

for each first write IOP (input/output operation) from a first compute node of the one or more compute nodes, executing, by a first storage node of the plurality of storage nodes, the each first write IOP with respect to a first copy of a first storage unit stored by the first storage node and referenced by the each first write IOP by—

assigning a first virtual block address (VBA) to a logical block address (LBA) referenced in the each first write IOP according to a first VBA counter;

incrementing the first VBA counter;

storing an association between the LBA and the first VBA;

writing data from the each first write IOP to a first physical storage location;

storing an association between the first physical storage location and the first VBA; and

transmitting the each first write IOP to a second storage node of the plurality of storage nodes with the first VBA.

11. The method of claim 10, further comprising, in response to failure of a first storage node of the plurality of storage nodes:

identifying, by the computing device, a current copy of a first storage unit of the one or more storage units assigned to the first storage node in the consistency database;

invoking, by the computing device, storage of another copy of the current copy on a different storage node of the plurality of storage nodes; and

updating, by the computing device, an entry in the consistency database for the different storage node to include the first unit of storage in the listing of the one or more storage units assigned to the each storage node.

12. The method of claim 10, further comprising, in response to detecting from the consistency database that a first copy of a first storage unit stored by a first storage node of the plurality of storage nodes is not current:

identifying, by the computing device, a current copy of the first storage unit in the listing of a second storage node of the plurality of storage nodes;

invoking, by the computing device, synchronization of the first copy with the current copy; and

updating, by the computing device, an entry in the consistency database for the first storage node to indicate that the first copy is current.

13. The method of claim 10, further comprising, in response to receiving an IOP (input/output operation) from a first compute node of the one or more compute nodes:

identifying, by a first compute node of the plurality of compute nodes, a storage unit referenced in the IOP;

identifying, by the computing device, a first storage node of the plurality of storage nodes that is listed in the consistency database as being a leader for that storage unit; and

transmitting, by the first compute node, a reference to the first storage node to the first compute node.

14. The method of claim 10, further comprising:

receiving, by a first storage node of the plurality of storage nodes, a write TOP (input/output operation) from a compute node of the one or more compute nodes;

executing, by the first storage node, the write IOP with respect to a first copy of a first storage unit stored by the each storage node;

transmitting, by the first storage node, the write IOP to a second storage node of the plurality of storage nodes; and

when no acknowledgment of the write IOP is received form the second storage node, transmitting, by the first storage node, a first notification of the notifications indicating that the second storage node is at least one of (c) not operational and (d) not current with respect to the first storage unit.

15. The method of claim 10, further comprising, for each second write IOP received from a different storage node of the plurality of storage nodes, executing, by the first storage node, the each second write IOP with respect to a second copy of a second storage unit stored by the first storage node and referenced by the each second write TOP by—

comparing a second VBA included with the each second write IOP to a second VBA counter;

when the second VBA is not consistent with the second VBA counter, notifying the different storage node that the first storage node is not current;

when the second VBA is consistent with the second VBA counter—

incrementing the second VBA counter;

storing an association between a second LBA included in the each second write IOP and the second VBA;

writing data from the each second write IOP to a second physical storage location;

storing an association between the second physical storage location and the VBA; and

transmitting an acknowledgment to the different storage node.

16. The method of claim 15, further comprising, for each second write IOP received from the different storage node of the plurality of storage nodes, executing, by the first storage node, the each second write IOP by:

when the second VBA is not consistent with the second VBA counter, refraining from executing the each second write IOP with respect to the second copy of the second storage unit referenced by the each second write IOP.

17. The method of claim 16, further comprising, when the second VBA is not consistent with the VBA counter:

identifying, by the first storage node, a range of VBAs according to a difference between the second VBA and the second VBA counter;

requesting and receiving, by the first storage node, third write IOPs including VBAs from the range of VBAs from the different storage node; and

executing, by the first storage node, each third write IOP and incrementing the VBA counter for each third write IOP.

18. The method of claim 10, further comprising, for each first write IOP received from the first compute node of the one or more compute nodes, executing, by the first storage node, the each first write IOP with respect to a first copy of a first storage unit stored by the each storage node and referenced by the each first write IOP by—

following execution of the each first write IOP, acknowledging execution of the each first write IOP to the first compute node only after acknowledging execution of all previously received IOPs referencing the first storage unit.