Data storage inventory indexing

• US 9,767,129 B2
• Filed: 02/17/2015
• Issued: 09/19/2017
• Est. Priority Date: 08/08/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Patiejunas, Kestutis, Donlan, Bryan J., Claiborn, Christian L.
• Primary Examiner(s): LEROUX, ETIENNE PIERRE
• Application Number:
  US14/624,528
• Publication Number:
  US 20150161184A1
• Time in Patent Office: 945 Days
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning

Abstract

Embodiments of the present disclosure are directed to, among other things, managing inventory indexing of one or more data storage devices. In some examples, a storage service may store an index associated with archived data. Additionally, the storage service may receive information associated with an operation performed on the archived data. The storage service may also partition the received information into subsets corresponding to an identifier. In some cases, the identifier may be received with or otherwise be part of the received information. The storage service may also retrieve at least a portion of the index that corresponds to the subset. Further, the storage service may update the retrieved portion of the index with at least part of the received information. The updating may be based at least in part on the subsets.

Claims

1. A computer-implemented method for index management, comprising:

under the control of one or more computer systems configured with executable instructions,

storing data in archival storage;

obtaining a data inventory index by at least;

if the data inventory index does not exist, generating the data inventory index associated with the stored data, the data inventory index including a data object identifier for at least a subset of parts of the stored data; and

if the data inventory index does exist, merging into the data inventory index at least a manifest of parts of the stored data;

selecting at least one of the parts of the manifest of parts that references a portion of the data inventory index to be updated based at least in part on the data object identifier for the at least one of the parts; and

updating a portion of the data inventory index based at least in part on the selected at least one part of the manifest of parts.

2. The computer-implemented method of claim 1, further comprising providing a tool to a remote computing device, the tool configured to enable the remote computing device to request the data inventory index associated with the stored data.

3. The computer-implemented method of claim 2, further comprising providing the data inventory index to the remote computing device in a representation of at least a predetermined plurality of available representations.

4. The computer-implemented method of claim 1, further comprising:

receiving a logical data container identifier corresponding with a completed job associated with the stored data; and

wherein selecting the at least one of the parts of the manifest of a parts that references the portion of the data inventory index to be updated is further based at least in part on the received logical data container identifier.

5. The computer-implemented method of claim 4, further comprising:

partitioning the selected parts of the manifest of parts based at least in part on the logical data container identifier; and

wherein merging the data inventory index includes at least adding the partitioned parts of the manifest of parts in chronological order, based at least in part on a time stamp of the data inventory index and a time stamp of the received logical data container corresponding to the completed job, when a data object identifier of the data inventory index matches a data object identifier of corresponding to the completed job.

6. The computer-implemented method of claim 1, further comprising:

retrieving the data inventory index; and

scheduling retrieval of data associated with at least one completed job corresponding to the data object identifier.

7. A system for index management, comprising:

one or more processors; and

memory with instructions that, when executed by the one or more processors, cause the system to;

obtain an index associated with archived data by at least;

if the index does not exist, generating the index; and

if the index does exist, merging into the index at least a list of parts associated with archived data;

store the index associated with archived data;

partition information associated with a completed operation performed on the archived data into one or more subsets;

retrieve at least a portion of the stored index corresponding to the one or more subsets; and

update the retrieved portion of the stored index with the partition information based at least in part on the one or more subsets.

8. The system of claim 7, wherein the index is generated based at least in part on first information associated with a first operation performed on the data.

9. The system of claim 8, wherein the index includes at least one of a time stamp associated with the first operation, a data object identifier associated with the first operation or a logical data container identifier associated with the first operation.

10. The system of claim 7, wherein the information associated with the operation performed on the data is received from a remote computing device configured to manage a storage device for storing the data, and wherein the storage device for storing the data is different from a second storage device for storing the index.

11. The system of claim 7, wherein the memory further includes instructions that, when executed by the one or more processors, cause the system to:

determine, based at least in part on the information associated with a completed operation performed on the archived data, a logical data container identifier; and

wherein partitioning the information associated into the one or more subsets is further based at least in part on the logical data container identifier.

12. The system of claim 11, wherein the memory further includes instructions that, when executed by the one or more processors, cause the system to perform statistical calculations based at least in part on at least one of how many bytes are included in the archived data, how old is the archived data, or how old is data of the archived data that was deleted.

13. A non-transitory computer-readable storage medium having stored thereon executable instructions that, when executed by one or more processors of a computer system, cause the computer system to at least:

obtain an index of operations associated with stored data by at least;

if the index of operations associated with stored data does not exist generating the index;

if the index does exist, merging into the index operations associated with the stored data;

store the index of operations associated with stored data, the index including at least one or more data object identifiers;

transmit at least one entry of the index, corresponding to at least one data object identifier of the data object identifiers, to a remote computing device;

receive information associated with the stored data corresponding to the at least one entry of the index; and

update the index based at least in part on the received information.

14. The non-transitory computer-readable storage medium of claim 13, wherein the instructions that cause the computer system to obtain the index further include instructions that cause the computer system to obtain the index from a storage device of a first computing device separate from a second storage device of a second computing device that stores the data.

15. The non-transitory computer-readable storage medium of claim 14, wherein the instructions further comprise instructions that, when executed by the one or more processors, cause the computer system to generate the index of operations based at least in part on information received from the second computing device.

16. The non-transitory computer-readable storage medium of claim 15, wherein the instructions further comprise instructions that, when executed by the one or more processors, cause the computer system to determine, based at least in part on the index, at least one of a time stamp associated with the index of operations or a data object identifier that indicates a logical mapping associated with storage of parts of the data.

17. The non-transitory computer-readable storage medium of claim 13, wherein the instructions that cause the computer system to update the index further include instructions that cause the computer system to update the index with information corresponding to operations performed on stored data based at least in part on a data object identifier of the data object identifiers.

18. The non-transitory computer-readable storage medium of claim 13, wherein the instructions that cause the computer system to update the index further include instructions that cause the computer system to update the index based at least in part on a batch schedule after at least one of a defined time or a defined number of operations to be performed on the stored data are queued.

19. The non-transitory computer-readable storage medium of claim 13, wherein the instructions further comprise instructions that, when executed by the one or more processors, cause the computer system to validate the index based at least in part on determining whether a requested data object identifier exists in the stored data.

20. The non-transitory computer-readable storage medium of claim 13, wherein the instructions further comprise instructions that, when executed by the one or more processors, cause the computer system to validate information associated with the index based at least in part on determining whether the received information matches the at least one entry of the index.