Service and APIs for remote volume-based block storage

• US 10,129,337 B2
• Filed: 04/24/2017
• Issued: 11/13/2018
• Est. Priority Date: 12/15/2011
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Lin, Yun, Sorenson, III, James Christopher, Salyers, David C.
• Primary Examiner(s): Hussain, Tauqir
• Assistant Examiner(s): Duong, Thao
• Application Number:
  US15/495,784
• Publication Number:
  US 20170230459A1
• Time in Patent Office: 568 Days
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0652 Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 63/06 for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/60 Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources admission control or resource allocation H04L47/70
• CPC Class Codes: H04L 69/04 Protocols for data compression, e.g. ROHC

Abstract

Methods, apparatus, and computer-accessible storage media for providing a volume-based block storage service and application programming interfaces (APIs) to the service. A block storage service and block storage service APIs may allow processes (applications or appliances) on the service client network to leverage remote, volume-based block storage provided by the service provider. The APIs may provide a standard interface to volume-based block storage operations on a remote data store. The service provider, the service clients, and/or third parties may develop various applications and/or appliances that may, for example, be instantiated in service clients'"'"' local networks and that leverage the block storage service via the APIs to create and manage volumes and snapshots on the remote data store and to upload and download data from the volumes and snapshots on the remote data store.

Claims

1. A method, comprising:

exposing, by a block storage service implemented on one or more devices within a provider network, block storage service application programming interfaces (APIs) to clients external to the provider network, wherein the provider network provides volume-based block storage to clients external to the provider network, wherein the block storage service APIs provide an application programming interface for volume-based block storage operations of the block storage service, and wherein the block storage service APIs include a create volume API, an upload block API, a create snapshot API, and a download block API; and

performing, by the block storage service;

receiving, from a client external to the provider network, a create volume request according to the create volume API;

creating, within the provider network, a volume according to the create volume API and according to one or more parameters of the create volume request;

returning, to the application, a create volume response according to the create volume API that indicates a volume identifier (ID) for the volume;

receiving, from the client external to the provider network, one or more upload block requests according to the upload block API, wherein each upload block request specifies the volume ID and data to be uploaded to the volume; and

for each of the one or more upload block requests, writing the respective data to the volume within the provider network according to the upload block API;

receiving, from the client external to the provider network, a create snapshot request according to the create snapshot API, wherein the create snapshot request specifies the volume ID;

creating, within the provider network, a snapshot of the volume within the provider network according to the create snapshot API;

receiving, from the client external to the provider network, one or more download block requests according to the download block API, wherein each download block request specifies data to be downloaded from the volume; and

for each of the one or more download block requests, reading the specified data from the volume within the provider network and returning, to the client external to the provider network, a download block response that includes the specified data according to the download block API.

2. The method as recited in claim 1, wherein a snapshot is a point-in-time capture of a volume, and wherein the method further comprises performing, by the block storage service:

subsequent to said creating, within the provider network, the snapshot of the volume within the provider network according to the create snapshot API, returning, to the application, a create snapshot response according to the create snapshot API that indicates a snapshot identifier (ID) for the snapshot;

receiving, from the application, one or more upload block requests according to the upload block API, wherein each upload block request specifies the snapshot ID and data to be uploaded to the snapshot; and

for each of the one or more upload block requests, writing the respective data as snapshot data for the respective snapshot according to one or more parameters of the upload block request.

3. The method as recited in claim 2, wherein the block storage service APIs further include a snapshot progress API, and wherein the method further comprises performing, by the block storage service, receiving, from the application, one or more snapshot progress notifications according to the upload block API, wherein each snapshot progress notification specifies a snapshot ID and an indication of how much of the respective snapshot has been uploaded or that the respective snapshot has been completed.

4. The method as recited in claim 1, wherein the block storage service APIs further include a delete volume API and a delete snapshot API, wherein a snapshot is a point-in-time capture of a volume, and wherein the method further comprises performing, by the block storage service:

receiving, from the application, either a delete volume request that specifies a volume to be deleted according to the delete volume API or a delete snapshot request that specifies a snapshot to be deleted according to the delete snapshot API; and

in response to the delete volume request or the delete snapshot request, deleting the respective volume or snapshot indicated by the request.

5. A system, comprising:

a plurality of computing devices on a provider network, each comprising at least one processor and memory, wherein the plurality of computing devices implement;

a block storage service of the provider network; and

application programming interfaces (APIs) to the block storage service for clients external to the provider network, wherein the APIs to the block storage service provide an application programming interface for the volume-based block storage operations of the block storage service, and wherein the APIs to the block storage service include a create volume API, an upload block API, a download block API, and a create snapshot API;

wherein the block storage service is configured to;

receive, from a client external to the provider network, one or more create volume requests according to the create volume API;

for at least one received create volume request;

create, within the provider network, a volume according to the create volume API and according to one or more parameters of the respective create volume request; and

return, to the respective client a create volume response according to the create volume API that indicates a volume identifier (ID) for the volume;

receive, from the client external to the provider network, one or more upload block requests according to the upload block API, wherein each upload block request specifies a volume ID and data to be uploaded to a respective volume; and

for at least one of the upload block requests, write the respective data to the respective volume within the provider network according to the upload block API.

6. The system as recited in claim 5, wherein the block storage service is further configured to, for the at least one received create volume request, create an initial snapshot for the respective volume, wherein a snapshot is a point-in-time capture of a volume.

7. The system as recited in claim 5, wherein the block storage service is further configured to, for the at least one received create volume request, create a volume manifest for the volume, wherein the volume manifest maps data blocks of the volume to locations of corresponding data chunks.

8. The system as recited in claim 7, wherein the block storage service is further configured to, for the at least one of the upload block requests, update the volume manifest of the respective volume according to the upload block request.

9. The system as recited in claim 5, wherein the block storage service is further configured to, for the at least one of the upload block requests, return, to the respective client, one or more data chunk tokens for the respective data written to the respective volume, wherein each data chunk token identifies a unit of the data that was uploaded to the respective volume according to the upload block request.

10. The system as recited in claim 5, wherein the block storage service is further configured to, for at least one other received create volume request:

determine that the other received create volume request is not valid according to one or more parameters of the respective create volume request; and

return, to the respective client, an indication of an error for the other received create volume request.

11. The system as recited in claim 5, wherein the block storage service is further configured to, for at least one other received upload block request:

determine that the other received upload block request is not valid according to one or more parameters of the respective upload block request; and

return, to the respective client, an indication of an error for the other received upload block request.

12. The system as recited in claim 5, wherein the block storage service is further configured to:

receive, from a client external to the provider network, one or more download block requests according to the download block API, wherein each download block request specifies a volume ID and volume data to be downloaded from the respective volume; and

for at least one of the download block requests, read the specified volume data according to one or more download block API parameters of the download block request and return, to the client, a download block response that includes the specified volume data according to the download block API.

13. The system as recited in claim 5, wherein a snapshot is a point-in-time capture of a volume, and wherein the block storage service is further configured to:

receive, from a client external to the provider network, one or more create snapshot requests according to the create snapshot API;

for at least one received create snapshot request;

create a snapshot according to one or more create snapshot API parameters of the respective create snapshot request; and

return, to the respective client, a create snapshot response according to the create snapshot API that indicates a snapshot identifier (ID) for the snapshot.

14. The system as recited in claim 13, wherein the block storage service is further configured to, for the at least one received create snapshot request, create a snapshot manifest for the snapshot, wherein the snapshot manifest maps data blocks of the snapshot to locations of data chunks stored.

15. The system as recited in claim 13, wherein the block storage service is further configured to:

receive, from at least one client external to the provider network, additional upload block requests according to the upload block API, wherein each additional upload block request specifies a snapshot ID and data to be uploaded to a respective snapshot; and

for at least one of the additional upload block requests, write the respective data as snapshot data for the snapshot according to one or more upload block API parameters of the respective upload block request.

16. The system as recited in claim 15, wherein the APIs to the block storage service further include a snapshot progress API for reporting progress of a snapshot being uploaded via the upload block API.

17. The system as recited in claim 5, wherein the APIs to the block storage service further include a delete volume API for requesting deletion of a specified volume and a delete snapshot API for requesting deletion of a specified snapshot, and wherein a snapshot is a point-in-time capture of a volume.

18. The system as recited in claim 5, wherein the block storage service further implements one or more techniques for reducing the amount of data that is uploaded for a volume by an application via the upload block API.

19. The system as recited in claim 18, wherein the one or more techniques include a data deduplication technique and a data compression technique.

20. A non-transitory computer-accessible storage medium storing program instructions computer-executable to:

implement a block storage service of a provider network; and

implement application programming interfaces (APIs) to the block storage service for clients external to the provider network, wherein the APIs to the block storage service provide an application programming interface for the volume-based block storage operations of the block storage service, and wherein the APIs to the block storage service include a create volume API, an upload block API, a download block API, and a create snapshot API;

receive, from a client external to the provider network, one or more create volume requests according to the create volume API;

for at least one received create volume request;

create, within the provider network, a volume according to the create volume API and according to one or more parameters of the respective create volume request; and

return, to the respective client a create volume response according to the create volume API that indicates a volume identifier (ID) for the volume;

receive, from the client external to the provider network, one or more upload block requests according to the upload block API, wherein each upload block request specifies a volume ID and data to be uploaded to a respective volume; and

for at least one of the upload block requests, write the respective data to the respective volume within the provider network according to the upload block API.