Storage scheme for a distributed storage system

• US 10,452,267 B2
• Filed: 09/13/2017
• Issued: 10/22/2019
• Est. Priority Date: 09/13/2017
• Status: Active Grant

• Current Assignee: Robin Systems, Inc. (Rakuten Group, Inc.)
• Original Assignee: Robin Systems, Inc. (Rakuten Group, Inc.)
• Inventors: Singh, Gurmeet, Venkatesan, Dhanashankar, Seetala, Partha Sarathi
• Primary Examiner(s): Choe, Yong J
• Application Number:
  US15/703,790
• Publication Number:
  US 20190079674A1
• Time in Patent Office: 769 Days
• CPC Class Codes: G06F 12/0253 Garbage collection, i.e. reclamation of unreferenced memory
• CPC Class Codes: G06F 2212/702 Conservative garbage collection
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A storage scheme allocates portions of a logical volume to storage nodes in excess of the capacity of the storage nodes. Slices of the storage nodes and segments of slices are allocated in response to write requests such that actual allocation on the storage nodes is only in response to usage. Segments are identified with virtual segment identifiers that are retained when segments are moved to a different storage node. Logical volumes may therefore be moved seamlessly to different storage nodes to ensure sufficient storage capacity. Data is written to new locations in segments having space and a block map tracks the last segment to which data for a given address is written. Garbage collection is performed to free segments that contain invalid data, i.e. data for addresses that have been subsequently written to.

Claims

1. A storage system comprising:

a storage node comprising a storage device, one or more processing devices, and one or more memory devices operably coupled to the one or more processing devices, the one or more memory devices storing executable code effective to cause the one or more processing devices to—

store a segment map including segment entries, each segment entry referencing a segment of a plurality of segments available for storage on the storage device, a storage volume to which the segment of the plurality of segments is allocated, and a virtual segment identifier;

store data in at least a portion of the plurality of segments such that each segment of the plurality of segments includes a first portion including data payloads from a plurality of write requests and a second portion including metadata entries, each metadata entry indicating a write address within the storage volume for one of the data payloads stored in the first portion;

for each write address included in the metadata entries of the segments of the plurality of segments—

mark as invalid all data payloads in the segments of the plurality of segments that have a corresponding metadata entry including the each write address except for a last-written data payload of a latest segment of the plurality of segments for the each write address, the latest segment for the each write address having a highest valued virtual segment identifier out of all the segments of the plurality of segments including the each write address in the metadata entries thereof.

2. The storage system of claim 1, wherein the executable code is further effective to cause the one or more processing devices to:

identify a portion of the plurality of segments for garbage collection;

for each segment of the plurality of segments in the portion—

identify valid data payloads in the first portion of the each segment that have not been marked as invalid;

write the valid data payloads and the metadata entries corresponding to the valid payloads in the second portion to a new segment of the plurality of segments;

change the segment entry for the each segment to indicate that it is free.

3. The storage system of claim 2, wherein the executable code is further effective to cause the one or more processing devices to identify the portion of the plurality of segments for garbage collection by determining that each segment of the portion of the plurality of segments has an amount of valid payloads that is below a threshold amount.

4. The storage system of claim 3, wherein the threshold amount is at least 40% of a storage capacity of the each segment.

5. The storage system of claim 1, wherein the executable code is further effective to cause the one or more processing devices to:

maintain a block map by, for each write request of a plurality of write requests, update an entry in the block map corresponding to a write address in the each write request to include a segment identifier of a segment of the plurality of segments to which a data payload from the each write request is written;

mark as invalid all data payloads in the plurality of segments for which the corresponding metadata entry including the each write address except for the last-written data payload of the latest segment for the each write address by marking as invalid all data payloads in segments other than a segment referenced by the entry in the block map corresponding to the each write address.

6. The storage system of claim 1, wherein the executable code is further effective to cause the one or more processing devices to:

maintain a block map by, for each write request of a plurality of write requests, update a block map entry in the block map corresponding to a write address in the each write request to include—

(a) a segment identifier of a segment of the plurality of segments to which a data payload from the each write request is written and

(b) an offset within the segment of the plurality of segments to which the data payload from the each write request was written;

for each write address included in the metadata entries of the plurality of segments, mark as invalid all data payloads in the plurality of segments for which the corresponding metadata entry includes the each write address except for the data payload at the offset included in the block map entry for the each write address within the segment referenced in the block map entry for the each write address.

7. The storage system of claim 1, wherein the first portion of the each segment begins at a first end of the each segment and the second portion of the each segment begins at a second end of the each segment opposite the first end.

8. The storage system of claim 7, wherein the executable code if further effective to cause the one or more processing devices to process the plurality of write requests by, for each write request of the plurality of write requests:

if a segment entry in the segment map references a storage volume referenced by the each write request and corresponds to an available segment, the available segment being a segment of the plurality of segments having sufficient space for the data payload of the each write request, (a) writing the data payload from the each write request at a free location at an end of the first portion of the available segment, (b) adding a new metadata entry to the second portion, the new metadata entry including a write address from the each write request, (c) adding a first block map entry to a block map, the block map entry mapping the write address from the each write request to the available segment and including an offset within the available segment to which the data payload of the each write request was written; and

if no segment entry in the segment map both references the storage volume referenced by the each write request and corresponds to a segment of the plurality of segments having sufficient space for the data payload of the each write request, (d) selecting an unassigned segment from the plurality of segments, (e) adding a reference to the storage volume referenced by the each write request to a segment entry of the segment map corresponding to the unassigned segment of the plurality of segments, (f) writing the data payload of the each write request to a free location at an end of the first portion of the unassigned segment, (g) adding the new metadata entry to the second portion of the assigned segment, (h) adding a second block map entry to the block map, the block map entry mapping the write address from the each write request to the assigned segment and including an offset within the assigned segment to which the data payload of the each write request was written.

9. The storage system of claim 8, wherein the executable code if further effective to cause the one or more processing devices to process a plurality of read requests by, for each read request of the plurality of read requests:

identifying a read block map entry in the block map corresponding to the each read request; and

reading read data from an offset of the read block map entry in a segment of the plurality of segments referenced by the read block map entry.

10. The storage system of claim 1, further comprising one or more application nodes in data communication with the storage node, the executable code further effective to cause the one or more processing devices to receive the plurality of write requests from the one or more application nodes.

11. A method comprising:

providing a storage node comprising a storage device, one or more processing devices, and one or more memory devices operably coupled to the one or more processing devices;

storing, by the storage node, a segment map including segment entries, each segment entry referencing a segment of a plurality of segments available for storage on the storage device, a storage volume to which the segment of the plurality of segments is allocated, and a virtual segment identifier;

storing, by the storage node, data in at least a portion of the plurality of segments such that each segment of the at least the portion of the plurality of segments includes a first portion including data payloads from a plurality of write requests and a second portion including metadata entries, each metadata entry indicating a write address within the storage volume for one of the data payloads stored in the first portion;

for each write address included in the metadata entries of the segments of the plurality of segments—

marking, by the storage node, as invalid all data payloads in the plurality of segments that have a corresponding metadata entry including the each write address except for a last-written data payload of a latest segment for the each write address, the latest segment for the each write address having a highest valued virtual segment identifier out of all the segments of the plurality of segments including the each write address in the metadata entries thereof.

12. The method of claim 11, further comprising:

identifying, by the storage node, a portion of the plurality of segments for garbage collection;

for each segment of the plurality of segments in the portion—

identifying, by the storage node, valid data payloads in the first portion of the each segment that have not been marked as invalid;

writing, by the storage node, the valid data payloads and the metadata entries corresponding to the valid payloads in the second portion to a new segment of the plurality of segments;

changing, by the storage node, the segment entry for the each segment to indicate that it is free.

13. The method of claim 12, further comprising identifying, by the storage node, the portion of the plurality of segments for garbage collection by determining that each segment of the portion of the plurality of segments has an amount of valid payloads that is below a threshold amount.

14. The method of claim 13, wherein the threshold amount is at least 40% of a storage capacity of the each segment.

15. The method of claim 11, further comprising:

maintaining, by the storage node, a block map by, for each write request of a plurality of write requests, updating an entry in the block map corresponding to a write address in the each write request to include a segment identifier of a segment of the plurality of segments to which a data payload from the each write request is written;

marking as invalid, by the storage node, all data payloads in the plurality of segments for which the corresponding metadata entry including the each write address except for the last-written data payload of the latest segment for the each write address by marking as invalid all data payloads in segments of the plurality of segments other than a segment referenced by the entry in the block map corresponding to the each write address.

16. The method of claim 11, further comprising:

maintaining, by the storage node, a block map by, for each write request of a plurality of write requests, updating a block map entry in the block map corresponding to a write address in the each write request to include—

(a) a segment identifier of a segment of the plurality of segments to which a data payload from the each write request is written and

(b) an offset within the segment of the plurality of segments to which the data payload from the each write request was written;

for each write address included in the metadata entries of the plurality of segments, marking, by the storage node, as invalid all data payloads in the plurality of segments for which the corresponding metadata entry includes the each write address except for the data payload at the offset included in the block map entry for the each write address within the segment of the plurality of segments referenced in the block map entry for the each write address.

17. The method of claim 11, wherein the first portion of the each segment begins at a first end of the each segment and the second portion of the each segment begins at a second end of the each segment opposite the first end.

18. The method of claim 17, further comprising processing, by the storage node, the plurality of write requests by, for each write request of the plurality of write requests:

in response to determining that a segment entry in the segment map references a storage volume referenced by the each write request and corresponds to an available segment, the available segment being a segment of the plurality of segments having sufficient space for the data payload of the each write request, performing—

(a) writing the data payload from the each write request at a free location at an end of the first portion of the available segment;

(b) adding a new metadata entry to the second portion, the new metadata entry including a write address from the each write request; and

(c) adding a first block map entry to a block map, the block map entry mapping the write address from the each write request to the available segment and including an offset within the available segment to which the data payload of the each write request was written.

19. The method of claim 18, further comprising processing a plurality of read requests by, for each read request of the plurality of read requests:

identifying, by the storage node, a read block map entry in the block map corresponding to the each read request; and

reading, by the storage node, read data from an offset of the read block map entry in a segment of the plurality of segments referenced by the read block map entry.

20. The method of claim 17, further comprising processing, by the storage node, the plurality of write requests by, for each write request of the plurality of write requests:

in response to determining that no segment entry in the segment map both references the storage volume referenced by the each write request and corresponds to a segment having sufficient space for the data payload of the each write request, performing—

(a) selecting an unassigned segment from the plurality of segments;

(b) adding a reference to the storage volume referenced by the each write request to a segment entry of the segment map corresponding to the unassigned segment;

(c) writing the data payload of the each write request to a free location at an end of the first portion of the unassigned segment;

(d) adding a new metadata entry to the second portion of the unassigned segment; and

(e) adding a second block map entry to the block map, the block map entry mapping the write address from the each write request to the unassigned segment and including an offset within the unassigned segment to which the data payload of the each write request was written.