Multi-tenant storage service object lifecycle management using transition job objects

• US 9,727,522 B1
• Filed: 12/14/2012
• Issued: 08/08/2017
• Est. Priority Date: 12/14/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Denny-Brown, II, Derek Ernest, Barber, Jeffrey Michael, Perry, Carl Yates, Elving, Christopher Henning, Gattu, Praveen Kumar
• Primary Examiner(s): Barry, Lance L
• Application Number:
  US13/715,844
• Time in Patent Office: 1,698 Days
• CPC Class Codes: G06F 9/5016 the resource being the memory
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

Methods and apparatus for a pipelined architecture for storage lifecycles are disclosed. A method includes identifying, during a transition candidate discovery iteration, storage objects of a multi-tenant service for which respective lifecycle transitions are to be scheduled, including a particular storage object identified as being a candidate for a particular lifecycle transition. The method includes generating transition job objects, each job comprising respective indications of one or more of the storage objects identified during the discovery iteration. The method includes assigning, during a transition execution iteration, resources to initiate storage operations for the particular lifecycle transition, and initiating, using the assigned resources, storage operations for the particular lifecycle transition.

Claims

1. A system, comprising:

one or more hardware computing devices comprising one or more respective hardware processors and memory and configured to implement;

a network-accessible multi-tenant storage service for storage objects owned by a plurality of clients; and

a lifecycle manager configured to;

identify, during a transition candidate discovery iteration, a set of storage objects of the storage service for which respective lifecycle transitions are to be scheduled, wherein at least one lifecycle transition of a particular storage object of the set comprises a transition from a first service level to a second service level;

store one or more transition job objects in the storage service, wherein each transition job object of the one or more transition job objects comprises an indication of one or more of the storage objects identified during the transition candidate discovery iteration, and wherein the particular storage object is indicated in a particular transition job object of the one or more transition job objects;

assign, during a transition execution iteration performed asynchronously with respect to the transition candidate discovery iteration, one or more resources to initiate storage transition operations in accordance with the particular transition job object, wherein the storage transition operations effect the lifecycle transition of the particular storage object from the first service level to the second service level; and

initiate, using the one or more resources, the storage transition operations in accordance with the particular transition job object.

2. The system as recited in claim 1, wherein the first service level differs from the second service level in at least one of:

a supported data durability level, a supported data availability level, a supported performance level, a supported version count, a storage object replication count, or a pricing policy.

3. The system as recited in claim 1, wherein the one or more hardware computing devices are further configured to:

initiate the transition candidate discovery iteration based at least in part on one or more of;

(a) a predetermined schedule, (b) a scheduled execution of a billing module configured to determine storage billing amounts for one or more clients of the storage service, (c) one or more notifications generated in accordance with lifecycle policies stored in the storage service, (d) one or more resource utilization metrics associated with the storage service, or (e) an indication of a status of a different transition execution iteration.

4. The system as recited in claim 1, wherein the one or more hardware computing devices are further configured to:

initiate the transition execution iteration based at least in part on one or more of;

(a) a predetermined schedule, (b) a determination that a number of transition job objects in the system exceeds a threshold value, (c) one or more resource utilization metrics associated with the storage service, or (e) an indication of a status of a different transition candidate discovery iteration.

5. The system as recited in claim 1, wherein a particular storage transition operation initiated in accordance with the particular transition job object comprise one or more of:

(a) an operation to repartition the particular storage object according to parameters of an erasure encoding policy associated with the second service level (b) an operation to copy at least a portion of the data of the particular storage object from a first storage device configured to remain powered on for continuous access, to a second storage device configured to remain in a low power state by default.

6. A method, comprising:

implementing, by one or more hardware computing devices comprising one or more respective hardware processors and memory, a network-accessible multi-tenant storage service;

identifying, by a lifecycle manager implemented by one or more hardware computing devices, during a transition candidate discovery iteration, a set of storage objects of the storage service for which respective lifecycle transitions are to be scheduled, including a particular storage object;

generating by the lifecycle manager, one or more transition job objects, wherein each transition job object of the one or more transition job objects comprises an indication of one or more of the storage objects identified during the transition candidate discovery iteration, and wherein the particular storage object is indicated in a particular transition job object of the one or more transition job objects as being a candidate for a particular lifecycle transition, wherein the particular lifecycle transition comprises a transition between a first service level and a second service level;

assigning, by the lifecycle manager, during a transition execution iteration, one or more resources to initiate storage transition operations for the particular transition job object, including a storage transition operation corresponding to the particular lifecycle transition of the particular storage object from the first service level to the second service level; and

initiating, using the one or more resources, the storage transition operations for the particular transition job object.

7. The method as recited in claim 6, wherein the first service level differs from the second service level in at least one of:

a supported data durability level, a supported data availability level, a supported performance level, a supported version count, a storage object replication count, or a pricing policy.

8. The method as recited in claim 6, further comprising:

initiating the transition candidate discovery iteration based at least in part on one or more of;

(a) a predetermined schedule, (b) a scheduled execution of a billing module configured to determine storage billing amounts for one or more clients of the storage service, (c) one or more notifications generated in accordance with lifecycle policies stored in the storage service, (d) one or more resource utilization metrics associated with the storage service, or (e) an indication of a status of a different transition execution iteration.

9. The method as recited in claim 6, further comprising:

initiating the transition execution iteration based at least in part on one or more of;

(a) a predetermined schedule, (b) a determination that a number of transition job objects exceeds a threshold value, (c) one or more resource utilization metrics associated with the storage service, or (e) an indication of a status of a different transition candidate discovery iteration.

10. The method as recited in claim 6, wherein a particular storage transition operation initiated in accordance with the particular transition job object comprises one or more of:

(a) an operation to repartition the particular storage object according to parameters of an erasure encoding policy or (b) an operation to copy at least a portion of the data of the particular storage object from a first storage device configured to remain powered on for continuous access, to a second storage device configured to remain in a low power state by default.

11. The method as recited in claim 6, wherein the transition execution iteration is asynchronous with respect to the transition candidate discovery iteration.

12. The method as recited in claim 6, further comprising:

storing, after said generating, the one or more transition job objects in the storage service.

13. The method as recited in claim 12, further comprising:

deleting, after determining that the storage transition operations for the particular transition job object have been performed, the particular transition job object from the storage service.

14. The method as recited in claim 6, wherein a particular resource of the one or more resources comprises a worker thread.

15. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more hardware processors implement a storage lifecycle transition job generator configured to:

perform a plurality of transition candidate discovery iterations, wherein each transition candidate discovery iteration of the plurality of transition candidate discovery iterations is initiated based on one or more scheduling criteria;

identify, during a particular transition candidate discovery iteration of the plurality of transition candidate discovery iterations, a set of storage objects of a multi-tenant storage service for which respective lifecycle transitions are to be scheduled, wherein the set of storage objects comprises a particular storage object identified as a candidate for a lifecycle transition from a first service level to a second service level;

generate one or more transition job objects, wherein one or more storage transition operations associated with one or more transition job objects effect the lifecycle transition of the particular storage object from the first service level to the second service level;

distribute indications of the set of storage objects among the one or more transition job objects; and

store the one or more transition job objects within one or more logical containers of the storage service.

16. The non-transitory computer-accessible storage medium as recited in claim 15, wherein the first service level differs from the second service level in at least one of:

a supported data durability level, a supported data availability level, a supported performance level, a supported version count, a storage object replication count, or a pricing policy.

17. The non-transitory computer-accessible storage medium as recited in claim 15, wherein a scheduling criterion of the one or more scheduling criteria is based at least in part on one or more of:

(a) an elapsed time since a previous transition candidate discovery iteration was initiated, (b) a scheduled execution of a billing module configured to determine storage billing amounts for one or more clients of the storage service, (c) one or more notifications generated in accordance with lifecycle policies stored in the storage service, (d) one or more resource utilization metrics associated with the storage service, or (e) an indication of a status of a transition execution iteration performed by a transition task dispatcher.

18. The non-transitory computer-accessible storage medium as recited in claim 15, wherein the lifecycle transition job generator is configured to:

determine a particular logical container of the one or more logical containers to store a particular transition job object based at least in part on a type of lifecycle transition to be scheduled for a storage object identified in the particular transition job object.