Lifecycle transition validation for storage objects

• US 10,853,337 B2
• Filed: 05/22/2017
• Issued: 12/01/2020
• Est. Priority Date: 12/14/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Barber, Jeffrey Michael, Gattu, Praveen Kumar, Denny-Brown, II, Derek Ernest, Elving, Christopher Henning, Perry, Carl Yates
• Primary Examiner(s): Barry, Lance Leonard
• Application Number:
  US15/602,017
• Publication Number:
  US 20170255589A1
• Time in Patent Office: 1,289 Days
• CPC Class Codes: G06F 16/219 Managing data history or versioning querying versioned data G06F16/2474; querying temporal data G06F16/2477
• CPC Class Codes: G06F 16/2474 Sequence data queries, e.g. querying versioned data
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Methods and apparatus for lifecycle support for storage objects are disclosed. A storage medium stores program instructions that when executed on a processor implement a storage lifecycle manager of a multi-tenant storage service. The lifecycle manager determines a lifecycle policy to be implemented for a storage object. The policy indicates lifecycle transitions of the object, including a transition from a first service level to a second service level. The first service level and the second service level may each specify a maximum number of versions of the storage object. The lifecycle manager initiates modification operations corresponding to the storage object'"'"'s data in accordance with the lifecycle policy, and verifies that the modification operations succeeded. The lifecycle manager provides, in response to a query, an indication of a current service level of the storage object.

Claims

1. A method, comprising:

determining a lifecycle policy for a storage object of a collection of storage objects of a storage service based at least in part on metadata for the collection indicating a policy assignment sequence number for the lifecycle policy and that the lifecycle policy applies to the storage object when stored as part of the collection, wherein the lifecycle policy comprises an indication of a lifecycle transition of the storage object, wherein the lifecycle transition comprises a transition from a first service level to a second service level, wherein the first service level and the second service level each, respectively, comprise at least one of;

a durability level,

an availability level,

a performance level indicating expected latencies for storage operations, or any combination thereof;

determining whether the lifecycle transition is valid based at least in part on a comparison of the policy assignment sequence number and a policy modification sequence number indicating a most recent modification of the lifecycle policy;

initiating, based at least in part on a determination that the lifecycle transition is valid, a modification operation associated with the storage object in accordance with the lifecycle policy;

verifying that the modification operation has succeeded; and

providing, in response to a query, an indication of a current service level of the storage object.

2. The method of claim 1, wherein the durability level indicates a probability of permanent data loss, and wherein the availability level indicates an amount of time which the storage object is expected to be inaccessible over a particular time period.

3. The method of claim 1, wherein the first service level and the second service level each further comprise a replication level, the method further comprising:

storing a first number of replicas of at least a portion of data of the storage object based on a first replication count corresponding to the replication level of the first service level; and

storing a second number of replicas of at least a portion of data of the storage object based on a second replication count corresponding to the replication level of the second service level, wherein the first number and the second number are different.

4. The method of claim 3, wherein the replicas comprise physical replicas of the storage object.

5. The method of claim 3, wherein the replicas comprise logical replicas of the storage object.

6. The method of claim 1, further comprising:

storing at least a portion of data of the storage object a first set of erasure coding parameters in accordance with a first set of coding parameters corresponding to the durability level of the first service level; and

storing the at least a portion of data of the storage object in accordance with a second set of erasure coding parameters corresponding to the durability level of the second service level.

7. The method of claim 1, wherein the performance level of the first service level further indicates an expected transition time for transitioning to the second service level.

8. A system, comprising:

one or more processors; and

a memory storing instructions that, when executed by the one or more processors,

cause the one or more processors to;

determine a lifecycle policy for a storage object of a collection of storage objects of a storage service based at least in part on metadata for the collection indicating a policy assignment sequence number for the lifecycle policy and that the lifecycle policy applies to the storage object when stored in the collection, wherein the lifecycle policy comprises an indication of a lifecycle transition of the storage object, wherein the lifecycle transition comprises a transition from a first service level to a second service level, wherein the first service level and the second service level each, respectively, comprise at least one of;

a durability level,

an availability level,

a performance level indicating expected latencies for storage operations, or

any combination thereof;

determine whether to perform a modification operation associated with the storage based at least in part on a comparison of the policy assignment sequence number and a policy modification sequence number indicated in metadata for the storage object;

initiate the modification operation associated with the storage object in accordance with the lifecycle policy in response to a determination that the policy assignment sequence number matches the policy modification sequence number;

verify that the modification operation has succeeded; and

provide, in response to a query, an indication of a current service level of the storage object.

9. The system of claim 8, wherein the instructions further cause the one or more processors to:

determine a data placement plan for at least a portion of data of the storage object in accordance with the lifecycle transition.

10. The system of claim 9, wherein the instructions further cause the one or more processors to:

determine the modification operation in accordance with the data placement plan.

11. The system of claim 8, wherein the instructions further cause the one or more processors to:

store a first number of replicas of at least a portion of data of the storage object based on a first replication count corresponding to the durability level of the first service level; and

store a second number of replicas of at least a portion of data of the storage object based on a second replication count corresponding to the durability level of the second service level.

12. The system of claim 8, wherein a transition criterion corresponding to the lifecycle transition is based at least in part on one or more of:

(a) an elapsed time since the storage object was created (b) an elapsed time since the storage object was last modified (c) an elapsed time since the storage object was last read (d) a frequency of access of the storage object, or (e) a cost of retaining the storage object in the first service level.

13. The system of claim 8, wherein the instructions further cause the one or more processors to:

generate, in response to a recommendation request indicating a client goal and a set of candidate storage objects, one or more lifecycle transition recommendations based at least in part on (a) the client goal and (b) the usage metrics associated with one or more candidate storage objects of the set.

14. A non-transitory, computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to:

determine a lifecycle policy for a storage object of a collection of storage objects of a storage service based at least in part on metadata for the collection indicating a policy assignment sequence number for the lifecycle policy and that the lifecycle policy applies to the storage object when stored in the collection, wherein the lifecycle policy comprises an indication of a lifecycle transition of the storage object, wherein the lifecycle transition comprises a transition from a first service level to a second service level, wherein the first service level and the second service level each, respectively, comprise at least one of;

a durability level indicating a probability of permanent data loss of the storage object,

an availability level indicating an amount of time which the storage object is expected to be inaccessible over a particular time period,

a performance level indicating expected latencies for storage operations,

a replication level indicating a number of replicas of the storage object to be stored, or

any combination thereof;

determine whether to perform a modification operation associated with the storage based at least in part on a comparison of the policy assignment sequence number and a policy modification sequence number indicated in metadata for the storage object;

initiate the modification operation associated with the storage object in accordance with the lifecycle policy in response to a determination that the policy assignment sequence number matches the policy modification sequence number;

verify that the modification operation has succeeded; and

provide, in response to a query, an indication of a current service level of the storage object.

15. The non-transitory, computer-readable storage medium of claim 14, wherein the instructions further cause the one or more processors to:

store a first number of replicas of at least a portion of data of the storage object based on a first replication count corresponding to the replication level of the first service level; and

store a second number of replicas of at least a portion of data of the storage object based on a second replication count corresponding to the replication level of the second service level.

16. The non-transitory, computer-readable storage medium of claim 14, wherein the performance level of the first service level further indicates response times for the storage operations.

17. The non-transitory, computer-readable storage medium of claim 14, wherein the instructions further cause the one or more processors to:

store at least a portion of data of the storage object a first set of erasure coding parameters in accordance with a first set of coding parameters corresponding to the first service level; and

store the at least a portion of data of the storage object in accordance with a second set of erasure coding parameters corresponding to the second service level.

18. The non-transitory, computer-readable storage medium of claim 14, wherein the instructions further cause the one or more processors to:

generate, in response to a recommendation request indicating a client goal and a set of candidate storage objects, one or more lifecycle transition recommendations based at least in part on (a) the client goal and (b) the usage metrics associated with one or more candidate storage objects of the set.

19. The non-transitory, computer-readable storage medium of claim 14, wherein the instructions further cause the one or more processors to:

divide data of the storage object into a plurality of fragments in accordance with the lifecycle transition; and

transform the plurality of fragments into a second plurality of fragments having a number of fragments different from the plurality of fragments.

20. The non-transitory, computer-readable storage medium of claim 19, wherein the instructions further cause the one or more processors to:

regenerate the data of the storage object based on a subset of the second plurality of fragments.