Service and APIs for remote volume-based block storage

• US 9,635,132 B1
• Filed: 12/15/2011
• Issued: 04/25/2017
• Est. Priority Date: 12/15/2011
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Lin, Yun, Sorenson, III, James Christopher, Salyers, David C.
• Primary Examiner(s): Gillis, Brian J
• Assistant Examiner(s): Duong, Thao
• Application Number:
  US13/327,605
• Time in Patent Office: 1,958 Days
• Field of Search: 709203, 709208, 709211, 709216-219, 709226, 709229, 709232
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0652 Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 63/06 for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/60 Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources admission control or resource allocation H04L47/70
• CPC Class Codes: H04L 69/04 Protocols for data compression, e.g. ROHC

Abstract

Methods, apparatus, and computer-accessible storage media for providing a volume-based block storage service and application programming interfaces (APIs) to the service. A block storage service and block storage service APIs may allow processes (applications or appliances) on the service client network to leverage remote, volume-based block storage provided by the service provider. The APIs may provide a standard interface to volume-based block storage operations on a remote data store. The service provider, the service clients, and/or third parties may develop various applications and/or appliances that may, for example, be instantiated in service clients'"'"' local networks and that leverage the block storage service via the APIs to create and manage volumes and snapshots on the remote data store and to upload and download data from the volumes and snapshots on the remote data store.

Claims

1. A method, comprising:

exposing, by a block storage service implemented on one or more devices within a provider network, block storage service application programming interfaces (APIs) to client applications external to the provider network, wherein the provider network provides volume-based block storage as a remote data store to client applications external to the provider network, wherein the block storage service APIs provide a standard application programming interface for volume-based block storage operations of the block storage service on the remote data store, and wherein the block storage service APIs that are exposed to client applications external to the provider network include a create volume API, an upload block API, a download block API, and a delete volume API; and

performing, by the block storage service;

receiving, from an application implemented on one or more devices on a client network, a create volume request according to the create volume API, wherein the create volume request received at the block storage service corresponds with a request generated by the application according to the create volume API;

creating, on the remote data store, a volume according to one or more parameters of the create volume request;

returning, to the application, a create volume response according to the create volume API that indicates a volume identifier (ID) for the volume;

receiving, from the application, one or more upload block requests according to the upload block API, wherein the one or more upload block requests received at the block storage service correspond with the one or more upload block requests generated by the application according to the upload block API, and wherein each upload block request specifies the volume ID and data to be uploaded to the volume; and

for each of the one or more upload block requests, writing the respective data to the remote data store as volume data for the volume according to one or more parameters of the upload block request;

receiving, from the application, one or more download block requests according to the download block API, wherein the one or more download block requests received at the block storage service correspond with the one or more download block requests generated by the application according to the download block API, and wherein each download block request specifies data to be downloaded from the volume;

for each of the one or more download block requests, reading the respective volume data from the remote data store according to one or more parameters of the download block request and returning, to the application, a download block response that includes the respective volume data according to the download block API;

receiving, from the application, a delete volume request that specifies a volume on the remote data store to be deleted according to the delete volume API; and

in response to the delete volume request, deleting the volume indicated by the delete volume request from the remote data store.

2. The method as recited in claim 1, wherein the block storage service APIs further include a create snapshot API, wherein a snapshot is a point-in-time capture of a volume, and wherein the method further comprises performing, by the block storage service:

receiving, from the application, a create snapshot request according to the create snapshot API;

creating, on the remote data store, a snapshot according to one or more parameters of the create snapshot request;

returning, to the application, a create snapshot response according to the create snapshot API that indicates a snapshot identifier (ID) for the snapshot;

receiving, from the application, one or more upload block requests according to the upload block API, wherein each upload block request specifies the snapshot ID and data to be uploaded to the snapshot; and

for each of the one or more upload block requests, writing the respective data to the remote data store as snapshot data for the respective snapshot according to one or more parameters of the upload block request.

3. The method as recited in claim 2, wherein the block storage service APIs further include a snapshot progress API, and wherein the method further comprises performing, by the block storage service, receiving, from the application, one or more snapshot progress notifications according to the upload block API, wherein each snapshot progress notification specifies a snapshot ID and an indication of how much of the respective snapshot has been uploaded or that the respective snapshot has been completed.

4. The method as recited in claim 1, wherein the block storage service APIs further include a delete snapshot API, wherein a snapshot is a point-in-time capture of a volume, and wherein the method further comprises performing, by the block storage service:

receiving, from the application, a delete snapshot request that specifies a snapshot on the remote data store to be deleted according to the delete snapshot API; and

in response to the delete snapshot request, deleting the snapshot indicated by the request from the remote data store.

5. A system, comprising:

a plurality of storage devices configured as a block-based data store on a provider network; and

a plurality of computing devices on the provider network, each comprising at least one processor and memory, wherein the plurality of computing devices implement a block storage service configured to perform volume based block storage operations on the data store;

wherein the plurality of computing devices expose application programming interfaces (APIs) to the block storage service to client networks of the provider network, wherein the APIs to the block storage service provide a standard application programming interface for the volume-based block storage operations of the block storage service for applications on the client networks external to the provider network, and wherein the APIs to the block storage service that are exposed to client applications external to the provider network include a create volume API, an upload block API, and a delete volume API for requesting deletion of a specified volume on the data store;

wherein the block storage service is configured to;

receive, from the applications on the client networks, create volume requests according to the create volume API, wherein the create volume requests received at the block storage service correspond with the create volume requests generated by the application according to the create volume API;

for at least one received create volume request;

create, on the data store, a volume according to one or more create volume API parameters of the respective create volume request; and

return, to the respective application, a create volume response according to the create volume API that indicates a volume identifier (ID) for the volume;

receive, from the applications on the client networks, upload block requests according to the upload block API, wherein the upload block requests received at the block storage service correspond with the upload block requests generated by the application according to the upload block API, and wherein each upload block request specifies a volume ID and data to be uploaded to a respective volume; and

for at least one of the upload block requests, write the respective data to the remote data store as volume data for the respective volume according to one or more upload block API parameters of the respective upload block request.

6. The system as recited in claim 5, wherein the block storage service is further configured to, for the at least one received create volume request, create an initial snapshot for the respective volume, wherein a snapshot is a point-in-time capture of a volume.

7. The system as recited in claim 5, wherein the block storage service is further configured to, for the at least one received create volume request, create a volume manifest for the volume, wherein the volume manifest maps data blocks of the volume to locations of data chunks stored on the data store.

8. The system as recited in claim 7, wherein the block storage service is further configured to, for the at least one of the upload block requests, update the volume manifest of the respective volume according to the upload block request.

9. The system as recited in claim 5, wherein the block storage service is further configured to, for the at least one of the upload block requests, return, to the respective application, one or more data chunk tokens for the volume data written to the data store, wherein each data chunk token identifies a unit of the data that was uploaded to the respective volume according to the upload block request.

10. The system as recited in claim 5, wherein the block storage service is further configured to, for at least one other received create volume request:

determine that the create volume request is not valid according to the one or more create volume API parameters of the respective create volume request; and

return, to the respective application, an indication of an error for the create volume request.

11. The system as recited in claim 5, wherein the block storage service is further configured to, for at least one other received upload block request:

determine that the upload block request is not valid according to the one or more upload block API parameters of the respective upload block request; and

return, to the respective application, an indication of an error for the upload block request.

12. The system as recited in claim 5, wherein the APIs to the block storage service further include a download block API, and wherein the block storage service is further configured to:

receive, from the applications on the client networks, download block requests according to the download block API, wherein each download block request specifies a volume ID and volume data to be downloaded from the respective volume; and

for at least one of the download block requests, read the specified volume data from the remote data store according to one or more download block API parameters of the download block request and return, to the application, a download block response that includes the specified volume data according to the download block API.

13. The system as recited in claim 5, wherein the APIs to the block storage service further include a create snapshot API, wherein a snapshot is a point-in-time capture of a volume, and wherein the block storage service is further configured to:

receive, from the applications on the client networks, create snapshot requests according to the create snapshot API;

for at least one received create snapshot request;

create, on the data store, a snapshot according to one or more create snapshot API parameters of the respective create snapshot request; and

return, to the respective application, a create snapshot response according to the create snapshot API that indicates a snapshot identifier (ID) for the snapshot.

14. The system as recited in claim 13, wherein the block storage service is further configured to, for the at least one received create snapshot request, create a snapshot manifest for the snapshot, wherein the snapshot manifest maps data blocks of the snapshot to locations of data chunks stored on the data store.

15. The system as recited in claim 13, wherein the block storage service is further configured to:

receive, from at least one of the applications on the client networks, additional upload block requests according to the upload block API, wherein each additional upload block request specifies a snapshot ID and data to be uploaded to a respective snapshot; and

for at least one of the additional upload block requests, write the respective data to the remote data store as snapshot data for the snapshot according to one or more upload block API parameters of the respective upload block request.

16. The system as recited in claim 15, wherein the APIs to the block storage service further include a snapshot progress API for reporting progress of a snapshot being uploaded to the data store via the upload block API.

17. The system as recited in claim 5, wherein the APIs to the block storage service further include a delete snapshot API for requesting deletion of a specified snapshot on the data store, wherein a snapshot is a point-in-time capture of a volume.

18. The system as recited in claim 5, wherein the block storage service further implements one or more techniques for reducing the amount of data that is uploaded for a volume by an application via the upload block API.

19. The system as recited in claim 18, wherein the one or more techniques include a data deduplication technique and a data compression technique.

20. A non-transitory computer-accessible storage medium storing program instructions computer-executable to:

implement a block storage service on one or more devices within a provider network, wherein the block storage service is configured to perform volume-based block storage operations on a data store of the provider network;

expose application programming interfaces (APIs) for the block storage service to client networks external to the provider network, wherein the APIs to the block storage service provide a standard application programming interface for the volume-based block storage operations of the block storage service for applications on the client networks, and wherein the APIs to the block storage service that are exposed to client applications external to the provider network include;

a create volume API for external applications to request creation of a volume on the data store according to one or more create volume API input parameters;

a create snapshot API for external applications to request creation of a snapshot on the data store according to one or more create snapshot API input parameters, wherein a snapshot is a point-in-time capture of a volume;

an upload block API for external applications to upload data blocks from the client networks to specified volumes or specified snapshots on the data store according to one or more upload block API input parameters;

a download block API for external applications to download specified data blocks from specified volumes or specified snapshots on the data store to the client networks according to one or more download block API input parameters; and

a delete volume API for requesting deletion of a specified volume on the data store.

21. The non-transitory computer-accessible storage medium as recited in claim 20, wherein the one or more create volume API input parameters include one or more authentication parameters for authenticating a caller with the block storage service, an optional volume size parameter for specifying a size for the volume to be created, and an optional snapshot identifier (ID) for specifying a snapshot from which the volume is to be created.

22. The non-transitory computer-accessible storage medium as recited in claim 20, wherein output parameters of the create snapshot API include a volume identifier (ID) for the created volume on the data store.

23. The non-transitory computer-accessible storage medium as recited in claim 20, wherein the one or more create snapshot API input parameters include one or more authentication parameters for authenticating a caller with the block storage service, a volume identifier (ID) that specifies a volume for which a snapshot is to be created, and a list of data chunk tokens, where each data chunk token identifies a unit of data in the respective volume.

24. The non-transitory computer-accessible storage medium as recited in claim 20, wherein output parameters of the create snapshot API include a snapshot identifier (ID) for the snapshot.

25. The non-transitory computer-accessible storage medium as recited in claim 20, wherein, for uploading data to a volume, the one or more upload block API input parameters include one or more authentication parameters for authenticating a caller with the block storage service, a volume identifier (ID) that specifies a volume to which data is being uploaded, a compressed parameter that indicates if the respective data is compressed, and, for each of one or more data blocks being uploaded, a data offset into the volume at which respective data is to be stored, a data length of the respective data, a checksum of the respective data, and the respective data.

26. The non-transitory computer-accessible storage medium as recited in claim 20, wherein, for uploading data to a snapshot, the one or more upload block API input parameters include one or more authentication parameters for authenticating a caller with the block storage service, a snapshot identifier (ID) that specifies a snapshot to which data is being uploaded, a data version parameter that specifies a version for the snapshot, a compressed parameter that indicates if the respective data is compressed, and, for each of one or more data blocks being uploaded, a data offset into the snapshot at which respective data is to be stored, a data length of the respective data, a checksum of the respective data, and the respective data.

27. The non-transitory computer-accessible storage medium as recited in claim 20, wherein output parameters of the upload block API include one or more data chunk tokens for the data uploaded to the data store according to the upload block API input parameters, wherein each data chunk token identifies a unit of the data that was uploaded to the data store according to the one or more upload block API input parameters.

28. The non-transitory computer-accessible storage medium as recited in claim 20, wherein, for downloading data from a volume, the one or more download block API input parameters include one or more authentication parameters for authenticating a caller with the block storage service, a volume identifier (ID) that specifies a volume from which data is to be downloaded, and, for each of one or more data blocks to be downloaded, a data offset parameter that indicates an offset into the volume from which data is to be downloaded, a data length parameter that indicates a length for the data to be downloaded from the volume at the data offset, and a compressed parameter that indicates if the respective data is to be compressed for downloading.

29. The non-transitory computer-accessible storage medium as recited in claim 20, wherein, for downloading data from a snapshot, the one or more download block API input parameters include one or more authentication parameters for authenticating a caller with the block storage service, a snapshot identifier (ID) that specifies a snapshot from which data is to be downloaded, and, for each of one or more data blocks to be downloaded, a data offset parameter that indicates an offset into the snapshot from which data is to be downloaded, a data length parameter that indicates a length for the data to be downloaded from the snapshot at the data offset, and a compressed parameter that indicates if the respective data is to be compressed for downloading.

30. The non-transitory computer-accessible storage medium as recited in claim 20, wherein output parameters of the download block API include, for each of one or more data blocks being downloaded, a data offset parameter that indicates an offset into the volume or snapshot for the respective data, a data length for the respective data, a checksum of the respective data, and the respective data.

31. The non-transitory computer-accessible storage medium as recited in claim 20, wherein the APIs to the block storage service further include a delete snapshot API for requesting deletion of a specified snapshot on the data store.

32. The non-transitory computer-accessible storage medium as recited in claim 20, wherein the APIs to the block storage service further include a snapshot progress API for reporting progress of a snapshot being uploaded to the data store via the upload block API.