Local storage clustering for redundancy coded data storage system

• US 10,061,668 B1
• Filed: 03/28/2016
• Issued: 08/28/2018
• Est. Priority Date: 03/28/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Lazier, Colin Laird, Brock, Adam Frederick
• Primary Examiner(s): Guyton, Philip A
• Application Number:
  US15/083,115
• Time in Patent Office: 883 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence

Abstract

A data transfer device is used to augment the capabilities of a data storage system. The data transfer device may be capable of persistently storing data for an indeterminate amount of time, and may be configured to store a portion of a bundle of redundancy coded shards that span between the data transfer device and a data storage system configured to store the remainder of the bundle. Data stored on the data transfer device may be read from and written directly to the data transfer device without transfer of data to the data storage system. If the data transfer device is not available, the remaining shards of the bundle may provide a regenerated, original form of the data.

Claims

1. A computer-implemented method, comprising:

configuring a data storage system to provision a cluster comprising a plurality of data transfer devices, the cluster being capable of providing a capability of the data storage system without an operable connection to the data storage system, the capability including processing of data storage requests by at least generating a bundle of redundancy coded shards from data received in association with the data storage requests;

processing, by the cluster, a first data storage request from a customer entity associated therewith, by at least;

generating, by the cluster, a bundle of redundancy coded shards from data associated with the first data storage request, the bundle including at least a plurality of identity shards, a first identity shard of the plurality of identity shards containing an original form of the data, and an encoded shard containing a redundancy coded form of the data, the bundle being configured such that a quorum quantity of shards of the bundle is sufficient to reconstruct, using a redundancy code, the original data with the bundle; and

storing the bundle on at least a subset of the plurality of data transfer devices of the cluster;

processing, by the cluster, a second data storage request to retrieve at least a portion of the data, by at least;

retrieving at least a portion of the first identity shard corresponding to the portion of the data requested; and

providing the portion of the data to a requestor associated with the second data storage request; and

in response to an event associated with the first identity shard, causing the first identity shard to be stored on durable storage of the data storage system, the cluster and the data storage system sharing a common application programming interface.

2. The computer-implemented method of claim 1, wherein the event is associated with a passage of a determined length of time after the first identity shard was generated by the cluster.

3. The computer-implemented method of claim 1, wherein the first identity shard is caused to be stored on the data storage system by at least providing a data transfer device associated with the first identity shard to an ingestion process associated with the data storage system.

4. The computer-implemented method of claim 1, wherein at least one of the first data storage request or the second data storage request is forwarded by the data storage system to the cluster for processing.

5. The computer-implemented method of claim 4, wherein the data transfer device is caused to transfer data associated with the first identity shard to the data storage system after the data transfer device is physically co-located with the data storage system.

6. The computer-implemented method of claim 1, wherein the data storage system is a first data storage system, and further comprising causing the data transfer device to transfer the first set of customer data to a second data storage system that is different than the first data storage system.

7. The computer-implemented method of claim 1, wherein at least one of the data transfer devices of the cluster is operably connected to the data storage system via a common network.

8. The computer-implemented method of claim 1, wherein the data transfer device associated with the first identity shard is a first data transfer device, and further comprising replacing the first data transfer device with a second data transfer device in response to the event.

9. The computer-implemented method of claim 1, further comprising, in response to a second event, causing the data storage system to expand the cluster by at least:

causing the cluster to configure the bundle to include an additional identity shard;

providing an additional data transfer device to a physical location associated with the data storage system;

configuring the cluster to further include the additional data transfer device; and

causing the cluster to use the redundancy code to recalculate the encoded shard based at least in part on additional data stored in the additional identity shard.

10. A system, comprising at least one computing device configured to implement one or more services, wherein the one or more services:

provision a cluster comprising a plurality of data transfer devices, the cluster providing a capability of the one or more services, the capability including data storage on a data storage system sharing a common application programming interface with the cluster, the plurality of data transfer devices usable for transferring data from a customer device to the system via an ingestion process, the customer device being remotely located relative to the system;

co-locate the cluster with the customer device;

cause the cluster to service a first set of requests requesting the capability directed to the cluster from the customer device; and

forward a second set of requests requesting the capability and received by the one or more services to the cluster for processing.

11. The system of claim 10, wherein the capability includes one or more capabilities associated with data storage.

12. The system of claim 10, wherein the one or more services are further configured to provision the cluster to provide an additional capability.

13. The system of claim 12, wherein the additional capability is a computational capability associated with the plurality of data transfer devices.

14. The system of claim 10, wherein the cluster is provisioned such that any transfer device of the plurality of data transfer devices is configured to respond to the first set of requests and the second set of requests on behalf of the cluster.

15. The system of claim 14, wherein the one or more services are further configured to cause a plural subset of the plurality of data transfer devices of the cluster to process one or more requests received by a data transfer device of the plurality of data transfer devices on behalf of the cluster.

16. A non-transitory computer-readable storage medium having stored thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to at least:

configure a plurality of data transfer devices to function as a cluster, the cluster providing one or more capabilities associated with a data storage system operably connected to the computer system, the cluster and the data storage system sharing a common application programming interface;

cause the cluster to be capable of providing the one or more capabilities to one or more requestors regardless of whether the cluster is operably connected to the data storage system;

cause the cluster to service requests from the requestor requesting the capability; and

if the cluster is incapable of servicing a portion of the requests, and if the cluster is operably connected to the data storage system, forward the portion of requests to the data storage system for further processing.

17. The non-transitory computer-readable storage medium of claim 16, wherein the one or more capabilities include data storage.

18. The non-transitory computer-readable storage medium of claim 16, wherein the cluster is co-located with the requestor and remotely from the data storage system.

19. The non-transitory computer-readable storage medium of claim 16, wherein the cluster is connected to the requestor via a first network and the requestor is connected to the data storage system via a second network that is not operably connected to the first network.