FILE STORAGE USING VARIABLE STRIPE SIZES

• US 20150277802A1
• Filed: 03/31/2014
• Published: 10/01/2015
• Est. Priority Date: 03/31/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: OIKARINEN, MATTI JUHANI, FRIGO, MATTEO, VINCENT, PRADEEP
• Granted Patent: US 9,772,787 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1474 in transactions G06F16/20 takes precedence
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0614 Improving the reliability of storage systems
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

A write request directed to a storage object is received at a distributed file storage service. Based on a variable stripe size selection policy, a size of a particular stripe of storage space to be allocated for the storage object is determined, which differs from the size of another stripe allocated earlier for the same storage object. Allocation of storage for the particular stripe at a particular storage device is requested, and if the allocation succeeds, the contents of the storage device are modified in accordance with the write request.

Claims

1. A system, comprising:

one or more computing devices configured to;

receive, at a multi-tenant storage service configured to distribute file contents across a plurality of storage devices, a write request directed to a file, wherein the write request indicates (a) a write offset within the file and (b) a write data payload;

determine, based at least in part on the write offset and the write data payload, that storage space is to be allocated to respond to the write request;

select, based at least in part on the write offset within the file, a size of a particular stripe of storage space to be allocated for the file;

identify, based at least in part on the size of the particular stripe, a storage subsystem node of the multi-tenant storage service at which at least one replica of the particular stripe is to be stored, wherein at least one replica of another stripe of the file is stored at a different storage subsystem node, and wherein the size of the particular stripe differs from a size of the other stripe;

allocate storage at the particular storage subsystem node for the particular stripe; and

modify contents of the particular stripe in accordance with the write request.

2. The system as recited in claim 1, wherein the other stripe starts at a smaller offset within the file than the write offset indicated in the write request, and wherein the size of the other stripe is smaller than the size of the particular stripe.

3. The system as recited in claim 1, wherein the size of the particular stripe is selected based at least in part on an analysis of metrics collected from a file system to which the file belongs.

4. The system as recited in claim 1, wherein the size of the particular block is determined based at least in part on a hint provided by a client.

5. The system as recited in claim 1, wherein the size of the particular stripe is determined based at least in part on a name of the file.

6. A method, comprising:

performing, by one or more computing devices;

receiving, at a distributed storage service, a write request directed to a file;

determining, based at least in part on a variable stripe size selection policy, a size of a particular stripe of storage space to be allocated for the file to accommodate the write request;

allocating storage for the particular stripe at a particular storage device, wherein storage for another stripe of the file is allocated at a different storage device, and wherein the other stripe has a different size than the particular stripe; and

modifying contents of the particular storage device in accordance with the write request.

7. The method as recited in claim 6, wherein in accordance with the variable stripe size selection policy, the size of the particular stripe is determined based at least in part on a write offset indicated in the write request.

8. The method as recited in claim 7, wherein the other stripe starts at a smaller offset within the file than the write offset indicated in the write request, and wherein the size of the other stripe is smaller than the size of the particular stripe.

9. The method as recited in claim 6, wherein in accordance with the variable stripe size selection policy, the size of the particular stripe is determined based at least in part on an analysis of metrics collected from a file system to which the file belongs.

10. The method as recited in claim 6, wherein in accordance with the variable stripe size selection policy, the size of the particular stripe is determined based at least in part on a hint provided by a client.

11. The method as recited in claim 6, wherein in accordance with the variable stripe size selection policy, the size of the particular stripe is determined based at least in part on a name of the file.

12. The method as recited in claim 6, further comprising:

allocating, in response to a triggering condition, storage space at a target storage device to store contents of the particular stripe and a different stripe of the file; and

combining contents of the particular stripe and the different stripe at the target storage device.

13. The method as recited in claim 12, wherein the triggering condition comprises a determination that the file has exceeded a threshold size.

14. The method as recited in claim 6, wherein the write request is formatted in accordance with one of:

(a) a version of NFS (Network File System), or (b) a version of SMB (Server Message Block).

15. The method as recited in claim 6, wherein the distributed storage service comprises a plurality of storage extents including a particular extent at the particular storage device, wherein each extent of the plurality of extents supports allocation of pages of one or more sizes, further comprising:

selecting, from among the plurality of storage devices, the particular storage device to allocate the particular stripe, based at least in part on a determination that the particular extent supports an allocation of a page of a particular size.

16. The method as recited in claim 6, wherein the file does not comprise any stripes prior to said receiving the write request, and wherein in accordance with the variable stripe size selection policy, the size of the particular stripe is determined based at least in part on a size of a write payload of the write request.

17. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors:

receive, at a distributed storage service, a write request directed to a storage object;

determine, based at least in part on a variable stripe size selection policy, a size of a particular stripe of storage space to be allocated for the storage object;

request an allocation of storage for the particular stripe at a particular storage device; and

initiate a modification of contents of the particular storage device in accordance with the write request.

18. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the variable stripe size selection policy comprises determining the size of the particular stripe based at least in part on a write offset indicated in the write request.

19. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the variable stripe size selection policy comprises determining the size of the particular stripe based at least in part on an analysis of metrics collected from a file system to which the storage object belongs.

20. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the storage object comprises one of:

a file, or a metadata structure.

21. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the instructions when executed on the one or more processors:

allocate, in response to a triggering condition, storage space at a target storage device to store contents of the particular stripe and a different stripe of the file; and

combine contents of the particular stripe and the different stripe at the target storage device.