Storage service lifecycle policy transition management

• US 9,355,060 B1
• Filed: 12/14/2012
• Issued: 05/31/2016
• Est. Priority Date: 12/14/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Barber, Jeffrey Michael, Gattu, Praveen Kumar, Perry, Carl Yates, Elving, Christopher Henning, Denny-Brown, Derek Ernest II
• Primary Examiner(s): Barry, Lance L
• Application Number:
  US13/715,850
• Time in Patent Office: 1,264 Days
• US Class Current: 1/1
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/5682 Policies or rules for updating, deleting or replacing the stored data

Abstract

Methods and apparatus for storage lifecycle configuration management are disclosed. A storage medium stores program instructions that, when executed on a processor, implement a storage lifecycle manager. The manager receives a lifecycle policy to be applied to storage objects of a container of a storage service. The policy indicates lifecycle transitions to be implemented for the objects. The manager stores (a) a policy modification sequence number (PMSN) indicative of a most recent modification of the policy and (b) a policy application sequence number (PASN) associated with the particular logical container, where the PASN is based on the PMSN. The manager later compares the PMSN and the PASN to determine whether a lifecycle transition is to be performed, and if the transition is to be performed, schedules one or more operations for the transition.

Claims

1. A system, comprising:

one or more hardware computing devices configured to;

implement a network-accessible multi-tenant storage service storing a plurality of storage objects in a plurality of logical containers;

receive a lifecycle policy to be applied to at least a set of storage objects stored within a particular logical container of the storage service, wherein the lifecycle policy indicates one or more lifecycle transitions to be implemented for the set of storage objects;

store, within the storage service, (a) a representation of the lifecycle policy in a data component of an additional storage object, and (b) a policy modification sequence number having a current value indicative of a most recent modification of the lifecycle policy in a metadata entry for the additional storage object;

store, within the storage service, an indication that the lifecycle policy applies to the set of storage objects, and a policy application sequence number in a metadata entry for the particular logical container, wherein the policy application sequence number is based at least in part on a particular value of the policy modification sequence number corresponding to a particular version of the lifecycle policy and is indicative of the particular version of the lifecycle policy;

determine, based at least in part on a comparison of the policy application sequence number and the policy modification sequence number, whether a modification operation is to be performed to implement a lifecycle transition for a particular storage object of the set of storage objects in accordance with the particular version of the lifecycle policy; and

in response to a determination that the modification operation is to be performed, schedule an execution of the modification operation.

2. The system as recited in claim 1, wherein the one or more computing devices are configured to determine that the modification operation is to be performed in response to a determination that the policy application sequence number is equal to the policy modification sequence number.

3. The system as recited in claim 1, wherein the additional storage object whose data component comprises the representation of the lifecycle policy is stored in a different logical container than the particular logical container.

4. The system as recited in claim 1, wherein a particular lifecycle transition indicated in the lifecycle policy comprises a transition between a first service level and a second service level, wherein the first service level differs from the second service level in at least one of:

a supported data durability level, a supported data availability level, a supported performance level, a supported version count, a storage object replication count, or a pricing policy.

5. The system as recited in claim 1, wherein the modification operation comprises one or more of:

(a) an operation to reparation the particular storage object according to parameters of an erasure coding policy, (b) an operation to copy at least a portion of data of the particular storage object from a first storage device configured to remain powered on for continuous access, to a second storage device configured to remain in a low power state by default, or (c) an operation to change a replication count of the particular storage object.

6. A method, comprising:

implementing a network-accessible multi-tenant storage service configured to store a plurality of storage objects in a plurality of logical containers;

receiving an indication of a lifecycle policy to be applied to at least a set of storage objects stored within a particular logical container of the storage service, wherein the lifecycle policy indicates one or more lifecycle transitions to be implemented for the set of storage objects;

storing, in the storage service, a representation of the lifecycle policy, and a policy modification sequence number having a current value indicative of a most recent modification of the lifecycle policy;

storing, in the storage service, a policy application sequence number indicative of an application of the lifecycle policy to the particular logical container, wherein the policy application sequence number is based at least in part on a particular value of the policy modification sequence number corresponding to a particular version of the lifecycle policy and is indicative of the particular version of the lifecycle policy;

determining, based at least in part on a comparison of the policy application sequence number and the policy modification sequence number, whether a lifecycle transition is to be performed in accordance with the particular version of the lifecycle policy; and

in response to determining that the lifecycle transition is to be performed, scheduling one or more operations to implement the lifecycle transition.

7. The method as recited in claim 6, further comprising:

determining that the lifecycle transition is to be performed in response to a determination that the policy application sequence number is equal to the policy modification sequence number.

8. The method as recited in claim 6, wherein the representation of the lifecycle policy is stored in an additional storage object created in response to said receiving.

9. The method as recited in claim 8, wherein the additional storage object is stored in a different logical container than the particular logical container.

10. The method as recited in claim 6, wherein a particular lifecycle transition indicated in the lifecycle policy comprises a transition between a first service level and a second service level, wherein the first service level differs from the second service level in at least one of:

a supported data durability level, a supported data availability level, a supported performance level, a supported version count, a storage object replication count, or a pricing policy.

11. The method as recited in claim 6, wherein a particular operation of the one or more operations comprises one or more of:

(a) an operation to repartition at least one of the set of storage objects according to parameters of an erasure encoding policy, (b) an operation to copy at least a portion of data of at least one of the set of storage objects from a first storage device configured to remain powered on for continuous access, to a second storage device configured to remain in a low power state by default, or (c) an operation to change a replication count of at least one of the set of storage objects.

12. The method as recited in claim 6, wherein said determining whether the lifecycle transition is to be performed in accordance with the lifecycle policy is performed by a job generator as part of a particular transition candidate discovery iteration of a plurality of transition candidate discovery iterations.

13. The method as recited in claim 12, further comprising:

initiating the particular transition candidate discovery iteration based at least in part on one or more of;

(a) a predetermined schedule, (b) a scheduled execution of a billing module configured to determine storage billing amounts for one or more clients of the storage service, (c) one or more notifications generated in accordance with lifecycle policies stored in the storage service, (d) one or more resource utilization metrics associated with the storage service, or (e) an indication of a status of a transition execution iteration performed by a transition task dispatcher.

14. The method as recited in claim 12, wherein said scheduling one or more operations to implement the lifecycle transition is performed by a transition task dispatcher as part of a particular transition execution iteration of a plurality of transition execution iterations, wherein the particular transition execution iteration is performed asynchronously with respect to the particular transition candidate discovery iteration.

15. The method as recited in claim 14, further comprising:

initiating the particular transition execution iteration based at least in part on one or more of;

(a) a predetermined schedule, (b) a determination that a number of transition job objects generated by the job generator exceeds a threshold value, (c) one or more resource utilization metrics associated with the storage service, or (e) an indication of a status of a transition candidate discovery iteration performed by the job generator.

16. The method as recited in claim 6, wherein said determining whether the lifecycle transition is to be performed in accordance with the lifecycle policy is performed by a task dispatcher as part of a particular transition execution iteration of a plurality of transition execution iterations.

17. The method as recited in claim 6, wherein said determining whether the lifecycle transition is to be performed in accordance with the lifecycle policy is performed by a worker thread assigned to the lifecycle transition by a task dispatcher during a particular transition execution iteration of a plurality of transition execution iterations.

18. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors implement a storage lifecycle manager configured to:

receive an indication of a lifecycle policy to be applied to at least a set of storage objects stored within a particular logical container of a multi-tenant storage service, wherein the lifecycle policy indicates one or more lifecycle transitions to be implemented for the set of storage objects;

store, in the storage service, a representation of the lifecycle policy, and a policy modification sequence number having a current value indicative of a most recent modification of the lifecycle policy;

store, in the storage service, a policy application sequence number associated with the particular logical container, wherein the policy application sequence number is based at least in part on a particular value of the policy modification sequence number corresponding to a particular version of the lifecycle policy and is indicative of the particular version of the lifecycle policy;

determine, based at least in part on a comparison of the policy application sequence number and the policy modification sequence number, whether a lifecycle transition is to be performed in accordance with the particular version of the lifecycle policy; and

in response to a determination that the lifecycle transition is to be performed, schedule one or more operations to implement the lifecycle transition.

19. The non-transitory computer-accessible storage medium as recited in claim 18, wherein the storage lifecycle manager is configured to determine that the lifecycle transition is to be performed in response to a determination that the policy application sequence number is equal to the policy modification sequence number.

20. The non-transitory computer-accessible storage medium as recited in claim 18, wherein said representation of the lifecycle policy is stored in a data component of an additional storage object created in response to said receiving.

21. The non-transitory computer-accessible storage medium as recited in claim 20, wherein the additional storage object is stored in a different logical container than the particular logical container.

22. The non-transitory computer-accessible storage medium as recited in claim 18, wherein a particular lifecycle transition indicated in the lifecycle policy comprises a transition between a first service level and a second service level, wherein the first service level differs from the second service level in at least one of:

a supported data durability level, a supported data availability level, a supported performance level, a supported version count, a storage object replication count, or a pricing policy.

23. The non-transitory computer-accessible storage medium as recited in claim 18, wherein a particular operation of the one or more operations comprises one or more of:

(a) an operation to repartition at least one of the set of storage objects according to parameters of an erasure encoding policy, (b) an operation to copy at least a portion of data of at least one of the set of storage objects from a first storage device configured to remain powered on for continuous access, to a second storage device configured to remain in a low power state by default, or (c) an operation to change a replication count of at least one of the set of storage objects.

24. The non-transitory computer-accessible storage medium as recited in claim 18, wherein the storage lifecycle manager is further configured to:

implement a programmatic interface enabling a client of the storage service to submit the indication of the lifecycle policy.