Data storage inventory indexing

• US 8,959,067 B1
• Filed: 08/08/2012
• Issued: 02/17/2015
• Est. Priority Date: 08/08/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Patiejunas, Kestutis, Donlan, Bryan J., Claiborn, Christian L.
• Primary Examiner(s): LEROUX, ETIENNE PIERRE
• Application Number:
  US13/569,665
• Time in Patent Office: 923 Days
• Field of Search: 707/2, 707/3, 707/696, 711/135, 718/100, 709204-206
• US Class Current: 707/696
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning

Abstract

Embodiments of the present disclosure are directed to, among other things, managing inventory indexing of one or more data storage devices. In some examples, a storage service may store an index associated with archived data. Additionally, the storage service may receive information associated with an operation performed on the archived data. The storage service may also partition the received information into subsets corresponding to an identifier. In some cases, the identifier may be received with or otherwise be part of the received information. The storage service may also retrieve at least a portion of the index that corresponds to the subset. Further, the storage service may update the retrieved portion of the index with at least part of the received information. The updating may be based at least in part on the subsets.

Claims

1. A computer-implemented method for index management, comprising:

under the control of one or more computer systems configured with executable instructions,

storing data in archival storage;

obtaining a data inventory index by at least;

if the data inventory index does not exist, generating the data inventory index associated with the stored data, the data inventory index including at least a manifest of parts of the stored data, the manifest including a data object identifier for at least a subset of the parts;

if the data inventory index does exist, merging into the data inventory index at least the manifest of parts of the stored data;

receiving information corresponding to a completed job associated with the stored data, the information including at least a logical data container identifier associated with the completed job;

partitioning the information based at least in part on the logical data container identifier to generate one or more logical data container queues corresponding to at least a subset of a plurality of logical data containers;

selecting at least one of the parts of the manifest that references a portion of the data inventory index to be updated based at least in part on the data object identifier for the at least one of the parts and the partitioned information; and

updating a portion of the data inventory index with at least a portion of the partitioned information based at least in part on the selected part of the manifest.

2. The computer-implemented method of claim 1, further comprising providing a tool to a remote computing device, the tool configured to enable the remote computing device to request the data inventory index associated with the stored data.

3. The computer-implemented method of claim 2, further comprising receiving, from the remote computing device and generated based at least in part on the tool, a request for the data inventory index.

4. The computer-implemented method of claim 3, further comprising providing the data inventory index to the remote computing device in a representation of at least a predetermined plurality of available representations.

5. The computer-implemented method of claim 1, wherein merging into the data inventory index at least the manifest of parts of the stored data includes merging the sorted data inventory index with the one or more logical data container queues includes at least adding the partitioned information in chronological order, based at least in part on a time stamp of the data inventory index and a time stamp of the received information corresponding to the completed job, when a data object identifier of the data inventory index matches a data object identifier of the received information corresponding to the completed job.

6. The computer-implemented method of claim 1, further comprising:

retrieving the data inventory index;

scheduling retrieval of data associated with at least one of the completed jobs corresponding to a data object identifier index part to identify data associated with at least one of the completed jobs; and

validating that the identified data associated with at least one of the completed jobs is stored in a data storage node of the archival storage.

7. A computer-implemented method for index management, comprising:

under the control of one or more computer systems configured with executable instructions,

generating an archival storage directory associated with stored data located in a memory by at least;

receiving information corresponding to a data operation associated with the stored data, the information including a data object identifier associated with the data operation and one or more logical data container identifiers associated with the data operation;

for at least a subset of the one or more logical data container identifiers, generating a table of parts; and

for at least a subset of the table of parts, storing the data object identifier;

obtaining an index by at least;

if the data inventory index does not exist, generating an index for the archival storage directory;

if the index does exist merging into the index at least the table of parts;

storing the archival storage directory, based at least in part on the index; and

validating that stored data identified by the data object identifier is stored in a data storage node of the memory.

8. The computer-implemented method of claim 7, further comprising:

receiving second information corresponding to a second data operation associated with the data or second data; and

generating a queue including the second information based at least in part on a logical data container associated with the second data operation.

9. The computer-implemented method of claim 8, further comprising:

retrieving at least a portion of the stored archival storage directory that corresponds to the queue of the second information; and

updating the portion of the stored archival storage directory with the received second information.

10. The computer-implemented method of claim 9, further comprising storing the updated portion of the archival storage directory.

11. The computer-implemented method of claim 9, wherein updating the portion of the archival storage directory includes at least merging the archival storage directory with the received second information.

12. The computer-implemented method of claim 7, wherein the data operation includes at least one of a data storage request, a stored data retrieval request, or a stored data deletion request.

13. A system for index management, comprising:

at least one memory that stores computer-executable instructions; and

at least one processor configured to access the at least one memory, wherein the at least one processor is configured to execute the computer-executable instructions to collectively at least;

obtain an index by at least;

if the index does not exist, generating the index;

if the index does exist merging into the index at least a list of parts associated with archived data;

store the index associated with the archived data;

receive information associated with a completed operation performed on the archived data;

partition the received information into one or more subsets based at least in part on a logical data container identifier of the received information;

retrieve at least a portion of the index corresponding to the one or more subsets; and

update the retrieved portion of the index with at least part of the received information based at least in part on the one or more subsets.

14. The system of claim 13, wherein the index is generated based at least in part on first information associated with a first operation performed on the data.

15. The system of claim 14, wherein the index includes at least one of a time stamp associated with the first operation, a data object identifier associated with the first operation or a logical data container identifier associated with the first operation.

16. The system of claim 13, wherein the information associated with the operation performed on the data is received from a remote computing device configured to manage a storage device for storing the data, and wherein the storage device for storing the data is different from a second storage device for storing the index.

17. The system of claim 16, wherein the at least one processor is further configured to execute the computer-executable instructions to collectively at least perform statistical calculations based at least in part on the index.

18. The system of claim 17, wherein the statistical calculations are based at least in part on at least one of how many bytes are included in the archived data, how old is the archived data, or how old is data of the archived data that was deleted.

19. One or more computer-readable media storing computer-executable instructions for index management that, when executed by one or more processors of a computer system, configure the computer system to perform operations comprising:

obtaining an index of operations associated with stored data by at least;

if the index of operations associated with stored data does not exist generating the index;

if the index does exist, merging into the index operations associated with the stored data;

storing the index, the index including at least one or more data object identifiers;

retrieving the data object identifiers of the index;

transmitting at least one entry of the index, corresponding to at least one of the data object identifiers, to a remote computing device;

in response to the transmitting the at least one entry of the index, receiving information associated with the stored data corresponding to the entry of the index; and

validating information associated with the index.

20. The one or more computer-readable media of claim 19, wherein the index is stored in a storage device of a first computing device different from storage device of a second computing device that stores the data.

21. The one or more computer-readable media of claim 20, wherein the instructions further configure the computer system to perform operations comprising generating the index of operations based at least in part on information received from the second computing device.

22. The one or more computer-readable media of claim 19, wherein the index further includes at least one of a time stamp associated with the operations associated with the stored data or a data object identifier that indicates a logical mapping associated with storage of parts of the data.

23. The one or more computer-readable media of claim 22, wherein the instructions further configure the computer system to perform operations comprising updating the index with new operation information based at least in part on the data object identifier associated with the new operation information.

24. The one or more computer-readable media of claim 23, wherein the updating is performed based at least in part on a batch schedule after at least one of a predefined time or a predefined number of operation updates are queued.

25. The one or more computer-readable media of claim 19, wherein validating the information associated with the index is based at least in part on determining whether a requested data object identifier exists in the stored data.

26. The one or more computer-readable media of claim 19, wherein validating the information associated with the index is based at least in part on determining whether the received information matches the entry of the index.