Data storage application programming interface

• US 10,698,880 B2
• Filed: 12/17/2015
• Issued: 06/30/2020
• Est. Priority Date: 08/08/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Patiejunas, Kestutis, Claiborn, Christian L., Lazier, Colin L., Suver Weir, Claire E., Seigle, Mark C.
• Primary Examiner(s): Lazaro, David R
• Assistant Examiner(s): Khurshid, Zia
• Application Number:
  US14/973,715
• Publication Number:
  US 20160103870A1
• Time in Patent Office: 1,657 Days
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/1443 Transmit or communication errors
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/219 Managing data history or versioning querying versioned data G06F16/2474; querying temporal data G06F16/2477
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: H04L 51/224 providing notification on incoming messages, e.g. pushed notifications of received messages

Abstract

An application programming interface for a data storage service provides a convenient mechanism for clients of the data storage service to access its various capabilities. An API call may be made to initiate a job and in response a job identifier may be provided. A separate API call specifying the job identifier may be made and a response providing information related to the job may result. Various API calls may be used to store data, retrieve data, obtain an inventory of stored data, and to obtain other information relating to stored data.

Claims

1. A computer-implemented method for an archival data storage system, comprising:

under the control of one or more computer systems of an archival data storage system;

receiving a retrieval request to retrieve a data object from persistent storage, where at least a portion of the data object is encoded based at least in part on a data encoding scheme to generate a set of encoded data components stored in individual data volume components of a set of data volume components, the retrieval request generating a data object identifier associated with the data object;

initiating a retrieval job to obtain the set of encoded data components from the set of data volume components in response to the retrieval request;

providing, to a computer system associated with the retrieval request, a retrieval job identifier associated with the retrieval job, where the retrieval job identifier comprises information that enables, when included in a status request, the computer system to provide status information associated with the retrieval job and, when included in a download request, to provide the data object;

erasure decoding the set of encoded data components to form a set of decoded data components;

combining the set of decoded data components to obtain the data object;

performing a data integrity check on the data object using payload integrity validation information encoded in the data object identifier;

receiving the download request encoding the retrieval job identifier; and

providing the data object for download in response to the download request.

2. The computer-implemented method of claim 1, further comprising:

receiving a status request encoding the retrieval job identifier; and

providing a current status of the retrieval job in response to the status request.

3. The computer-implemented method of claim 1, wherein:

the download request includes information indicating a particular portion of the data object; and

providing the data object for download includes providing the particular portion of the data object.

4. The computer-implemented method of claim 3, wherein providing the particular portion of the data object further comprises providing a data verification value for the particular portion, where the data verification value usable to determine whether, when downloaded, the particular portion is corrupted.

5. The computer-implemented method of claim 1, wherein the data encoding scheme is a redundancy coding scheme.

6. A system comprising:

one or more processors; and

memory with instructions that, as a result of being executed by the one or more processors, cause the system to;

receive a first request including a first identifier associated with a data object maintained by a storage system, where at least a portion of the data object is encoded based at least in part on a data encoding scheme to generate a set of encoded data components, wherein an encoded data component of the set of encoded data components is stored in a data volume component of a set of data volume components;

initiate a job to retrieve the set of encoded data components from the set of data volume components;

receive a second request including a second identifier, where the second identifier is associated with the job and, when included in the second request, causes the system to provide information associated with the job;

provide information associated with the job in response to the second request;

erasure decode the set of encoded data components to form a set of decoded data components;

combine the set of decoded data components to obtain the data object;

perform a data integrity check on the data object using payload integrity validation information encoded in the first identifier;

receive a third request including the second identifier, where the second identifier is further to, when included in the third request, cause the system to provide the data object; and

provide the data object in response to the third request.

7. The system of claim 6, wherein the first identifier and the second identifier are different.

8. The system of claim 6, wherein the memory further includes instructions that, as a result of being executed by the one or more processors cause the system to, as a result of the second request including a notification parameter, provide a notification upon completion of the job prior to receiving the third request.

9. The system of claim 6, wherein providing the data object further comprises copying at least a portion of the data object to a storage device of the storage system accessible to an entity associated with the first request.

10. The system of claim 6, wherein:

the data object is associated with a logical data container; and

the first identifier is an identifier for the logical data container.

11. The system of claim 6, wherein providing information associated with the job further comprises providing a status of the job.

12. The system of claim 6, wherein providing information associated with the job further comprises providing a log of one or more actions performed in connection with the data object.

13. The system of claim 6, wherein:

the first request originated from a customer computer system of a customer of the storage system;

the first request includes metadata provided by the customer computer system; and

the information associated with the job includes at least a portion of the metadata.

14. A non-transitory computer-readable storage medium storing thereon instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to:

receive a first electronic message comprising a first identifier corresponding to data stored by a storage system, where at least a portion of the data is encoded based at least in part on a data encoding scheme to generate a set of encoded data components stored in individual data volume components of a set of data volume components;

initiate a job in connection with the set of encoded data components, where a second identifier is associated with the job and useable to retrieve information associated with the job and the data stored by the storage system;

erasure decode the set of encoded data components to form a set of decoded data components;

combine the set of decoded data components to obtain the data;

perform a data integrity check on the data using payload integrity validation information encoded in the first identifier;

receive a second electronic message comprising the second identifier;

as a result of receipt of the second electronic message, determine the data stored by the storage system is associated with the job; and

provide the data stored by the storage system.

15. The non-transitory computer-readable storage medium of claim 14, wherein the first identifier and the second identifier are different.

16. The non-transitory computer-readable storage medium of claim 14, wherein:

the data comprises one or more data objects associated with a logical data container; and

the first identifier identifies the logical data container.

17. The non-transitory computer-readable storage medium of claim 16, wherein the instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to:

receive a third electronic message comprising the second identifier; and

provide, in response to the third electronic message, information associated with the job, where the information associated with the job includes a collection of one or more identifiers that each identifies at least one of the one or more data objects.

18. The non-transitory computer-readable storage medium of claim 16, wherein:

the first electronic message originated from a customer computer system of a customer of the storage system;

the first electronic message includes metadata provided by the customer computer system; and

the information associated with the job includes at least a portion of the metadata.

19. The non-transitory computer-readable storage medium of claim 18, wherein the instructions that cause the computer system to provide the data stored by the computer system further include instructions that cause the computer system to provide the data stored by the storage system by at least creating a copy of the data stored by the storage system on a second storage device of the storage system accessible to the customer computer system.

20. The non-transitory computer-readable storage medium of claim 19, wherein the information associated with the job indicates whether the copy of the data stored by the storage system has been stored in the second storage device of the storage system.

21. The non-transitory computer-readable storage medium of claim 14, wherein the instructions, as a result of being executed by the one or more processors, cause the computer system to provide a notification of completion of the job.