Archival data identification

• US 9,213,709 B2
• Filed: 08/08/2012
• Issued: 12/15/2015
• Est. Priority Date: 08/08/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Patiejunas, Kestutis, Donlan, Bryan J., Lazier, Colin L., Sorenson, James Christopher III, Seigle, Mark C., Claiborn, Christian L.
• Primary Examiner(s): Filipczyk, Marc
• Application Number:
  US13/569,994
• Publication Number:
  US 20140046906A1
• Time in Patent Office: 1,224 Days
• Field of Search: 707/661, 707/690, 707/694, 707/696, 707/701, 709204-206, 709/226, 711/100, 711/135, 711/163, 711/165, 711/166
• US Class Current: 1/1
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 21/606 by securing the transmission between two devices or processes
• CPC Class Codes: G06F 21/6209 to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Methods and systems are provided herein that facilitate cost-effective and reliable data identification in an archival data storage system. In an embodiment, a data object stored in an archival data storage system is identified by a data object identifier which encodes storage location information that may be used to locate a data object stored in an archival data storage system that reduces the cost to store a global index. The data object identifier may also encode policy information such as access control information usable for validating subsequent access to the data object, payload validation information such as size and digest usable for verifying the integrity of the payload data, metadata validation information such as error-detection codes usable for validating integrity of the data object identifier and other information.

Claims

1. A computer-implemented method, comprising:

under the control of one or more computer systems configured with executable instructions,

receiving, from a requesting entity, a data store request to store a data object;

storing the data object at a transient data store;

prior to storing the data object in a non-transient data storage location, providing, to the requesting entity, a data object identifier for the data object, the data object identifier encoding at least storage location information used to indicate the non-transient data storage location, payload validation information, and policy information;

storing the data object in the non-transient data storage location;

receiving a data retrieval request to retrieve the data object, the data retrieval request specifying the data object identifier;

decoding the data object identifier to obtain at least the storage location information, the payload validation information, and the policy information;

validating the data retrieval request using at least the policy information;

retrieving the data object as a retrieved data object from the non-transient data storage location using at least the storage location information;

validating integrity of the retrieved data object using at least the payload validation information; and

providing the retrieved data object.

2. The computer-implemented method of claim 1, wherein the storage location information includes a reference to a hierarchical data structure in which the data object is stored.

3. The computer-implemented method of claim 1, where the payload validation information includes at least one of timestamp, size, or digital digest of at least a portion of the data object.

4. The computer-implemented method of claim 1, wherein the policy information includes at least access control information.

5. The computer-implemented method of claim 1, wherein the data object identifier further encodes metadata validation information usable to validate integrity of information encoded by the data object identifier.

6. The computer-implemented method of claim 5, wherein the metadata validation information includes at least an error-detection code.

7. The computer-implemented method of claim 1, wherein providing the data object identifier includes encrypting at least the storage location information, the payload validation information, and the policy information using a cryptographic key.

8. The computer-implemented method of claim 1, further comprising storing the data object identifier with the data object.

9. The computer-implemented method of claim 8, wherein retrieving the data object includes verifying the data object identifier that is stored with the data object.

10. The computer-implemented method of claim 1, wherein the storage location information encodes at least a volume identifier and an object identifier, the volume identifier indicating a volume at which the data object is stored and the object identifier usable for identifying a location within the volume at which the data object is stored.

11. The computer-implemented method of claim 1, wherein the validation information includes at least payload validation information and wherein performing validation includes validating integrity of the retrieved data object using at least the payload validation information.

12. The computer-implemented method of claim 11, wherein the payload validation information includes at least a cryptographic hash of at least a portion of the data object.

13. The computer-implemented method of claim 1, wherein performing validation includes determining whether access to the data object is permitted based at least in part on the policy information.

14. An archival data storage system, comprising:

one or more processors; and

memory, including instructions executable by the one or more processors to cause a computer system to at least;

receive, from a requesting entity, a data store request to store a data object;

store the data object at a transient data store;

prior to storing the data object in a non-transient data storage location, provide, to the requesting entity, a data object identifier for the data object, the data object identifier encoding at least storage location information used to indicate the non-transient data storage location, payload validation information, and policy information;

store the data object in the non-transient data storage location;

receive a data retrieval request to retrieve the data object, the data retrieval request specifying the data object identifier;

decode the data object identifier to obtain at least the storage location information, the payload validation information, and the policy information;

validate the data retrieval request using at least the policy information;

retrieve the data object as a retrieved data object from the non-transient data storage location using at least the storage location information;

validate integrity of the retrieved data object using at least the payload validation information; and

provide the retrieved data object.

15. The system of claim 14, wherein the instructions executable by the one or more processors further cause the computer system to provide at least one application programming interface for performing a plurality of data operations, the plurality of data operations including at least data storage, data retrieval, data deletion, and metadata query.

16. The system of claim 14, wherein:

the data store request is one or more requests to store the data object, each of the one or more requests specifying at least a portion of the data object; and

the instructions executable by the one or more processors further cause the computer system to;

provide at least one application programming interface for performing a plurality of data operations;

receive, through the application programming interface, the one or more requests to store the data object; and

provide, through the application programming interface, the data object identifier.

17. The system of claim 16, wherein at least some of the validation information is based at least in part on the one or more requests to store the data objects.

18. The system of claim 16, wherein the data object identifier is associated with the portions of the data object collectively specified by the one or more requests.

19. The system of claim 14, where the instructions that cause the computer system to decode the data object identifier includes instructions that cause the computer system to decrypt at least a portion of the data object identifier using a cryptographic key.

20. One or more non-transitory computer-readable storage media having collectively stored thereon executable instructions that, when executed by one or more processors of a computing resource provider'"'"'s computer system, cause the computer system to at least:

receive, from a requesting entity, a data store request to store a data object;

store the data object at a transient data store;

prior to storing the data object in a non-transient data storage location, provide, to the requesting entity, a data object identifier for the data object, the data object identifier encoding at least storage location information used to indicate the non-transient data storage location, payload validation information, and policy information;

store the data object in the non-transient data storage location;

receive a data retrieval request to retrieve the data object, the data retrieval request specifying the data object identifier;

decode the data object identifier to obtain at least the storage location information, the payload validation information, and the policy information;

validate the data retrieval request using at least the policy information;

retrieve the data object as a retrieved data object from the non-transient data storage location using at least the storage location information;

validate integrity of the retrieved data object using at least the payload validation information; and

provide the retrieved data object.

21. The computer-readable storage media of claim 20, wherein the data object identifier encodes user-provided information.

22. The computer-readable storage media of claim 20, wherein the data object identifier is generated in response to a request to store the data object.

23. The computer-readable storage media of claim 20, wherein:

the executable instructions that cause the computer system to retrieve the data object cause the computer system to retrieve the data object as a plurality of data components; and

the executable instructions further include executable instructions that cause the computer system to restore the data object at least by;

causing application of one or more decoding schemes corresponding to one or more encoding schemes to the plurality of data components, the plurality of data components being generated by causing application of the one or more encoding schemes to the data object; and

combining one or more data parts resulting from the application of the one or more decoding schemes corresponding to the one or more encoding schemes to the plurality of data components.

24. The computer-readable storage media of claim 23, wherein the executable instructions that cause the computer system to restore the data object further include executable instructions that cause the computer system to restore the data object by combining one or more data parts that are retrievable based at least in part on the data object identifier.

25. The computer-readable storage media of claim 20, wherein the payload validation information includes data object integrity validation information for verifying integrity of the retrieved data object, data authorization information for verifying data access rights associated with the data object, and metadata validation information for verifying integrity of information included in the data object identifier.

26. The computer-readable storage media of claim 20, wherein the executable instructions, when executed by the one or more processors of the computing resource provider'"'"'s computer system, further cause the computer system to:

receive a request to delete the data object, the request specifying the data object identifier;

validate the request using at least the payload validation information;

locate the data object using at least the storage location information; and

delete the data object.