Storage lifecycle pipeline architecture

• US 10,642,654 B2
• Filed: 08/07/2017
• Issued: 05/05/2020
• Est. Priority Date: 12/14/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Barber, Jeffrey Michael, Denny-Brown, II, Derek Ernest, Perry, Carl Yates, Elving, Christopher Henning, Gattu, Praveen Kumar
• Primary Examiner(s): Barry, Lance Leonard
• Application Number:
  US15/671,099
• Publication Number:
  US 20170337154A1
• Time in Patent Office: 1,002 Days
• CPC Class Codes: G06F 9/5016 the resource being the memory
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

Methods and apparatus for a pipelined architecture for storage lifecycles are disclosed. A method includes identifying, during a transition candidate discovery iteration, storage objects of a multi-tenant service for which respective lifecycle transitions are to be scheduled, including a particular storage object identified as being a candidate for a particular lifecycle transition. The method includes generating transition job objects, each job comprising respective indications of one or more of the storage objects identified during the discovery iteration. The method includes assigning, during a transition execution iteration, resources to initiate storage operations for the particular lifecycle transition, and initiating, using the assigned resources, storage operations for the particular lifecycle transition.

Claims

1. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors implement a storage lifecycle task dispatcher configured to:

perform a plurality of lifecycle transition execution iterations for storage objects of a multi-tenant storage service, wherein each lifecycle transition execution iteration of the plurality of lifecycle transition execution iterations is initiated based on one or more scheduling criteria;

identify, during a particular lifecycle transition execution iteration of the plurality of lifecycle transition execution iterations, a set of transition job objects to be validated for execution;

validate a particular transition job object of the set; and

in response to validating the particular transition job object of the set;

assign one or more resources to initiate operations to implement lifecycle transitions indicated in the particular transition job object; and

monitor a completion state of an operation to which a particular resource of the one or more resources is assigned.

2. The non-transitory computer-accessible storage medium as recited in claim 1, wherein a scheduling criterion of the one or more scheduling criteria is based at least in part on one or more of:

(a) a predetermined schedule, (b) a determination that a number of transition job objects stored in the storage service exceeds a threshold value, (c) one or more resource utilization metrics associated with the storage service, or (e) an indication of a status of a transition candidate discovery iteration performed by a transition job generator.

3. The non-transitory computer-accessible storage medium as recited in claim 1,

wherein the one or more resources comprise a worker thread, and

wherein the worker thread is configured to:

subsequent to completion of a successful operation, send a task-done notification to the dispatcher, and

for an unsuccessful operation, send a task-failed notification to the dispatcher.

4. The non-transitory computer-accessible storage medium as recited in claim 1, wherein a particular operation initiated to implement a lifecycle transition indicated in the particular transition job object comprises one or more of:

(a) an operation to repartition a storage object according to parameters of an erasure encoding policy or (b) an operation to copy at least a portion of the data of a storage object from a first storage device configured to remain powered on for continuous access, to a second storage device configured to remain in a low power state by default.

5. The non-transitory computer-accessible storage medium as recited in claim 1,

wherein the storage objects of the plurality of lifecycle transition execution iterations are determined by a transition candidate discovery iteration;

and,

wherein said perform the plurality of transition execution iterations is performed asynchronously with respect to the transition candidate discovery iteration.

6. The non-transitory computer-accessible storage medium as recited in claim 1, wherein the program instructions, when executed on the one or more processors further perform delete, after a determination that the storage operations for the particular transition job object have been performed, the particular transition job object from the storage service.

7. A system, comprising:

one or more hardware computing devices comprising one or more respective hardware processors and memory and configured to implement a storage lifecycle task dispatcher configured to;

perform a plurality of lifecycle transition execution iterations for storage objects of a multi-tenant storage service, wherein each lifecycle transition execution iteration of the plurality of lifecycle transition execution iterations is initiated based on one or more scheduling criteria;

identify, during a particular lifecycle transition execution iteration of the plurality of lifecycle transition execution iterations, a set of transition job objects to be validated for execution;

validate a particular transition job object of the set; and

in response to the validation of the particular transition job object of the set;

assign one or more resources to initiate operations to implement lifecycle transitions indicated in the particular transition job object; and

monitor a completion state of an operation to which a particular resource of the one or more resources is assigned.

8. The system as recited in claim 7, wherein a scheduling criterion of the one or more scheduling criteria is based at least in part on one or more of:

(a) a predetermined schedule, (b) a determination that a number of transition job objects stored in the storage service exceeds a threshold value, (c) one or more resource utilization metrics associated with the storage service, or (e) an indication of a status of a transition candidate discovery iteration performed by a transition job generator.

9. The system as recited in claim 7, wherein the one or more resources comprise a worker thread.

10. The system as recited in claim 7, wherein a particular operation initiated to implement a lifecycle transition indicated in the particular transition job object comprises one or more of:

(a) an operation to repartition a storage object according to parameters of an erasure encoding policy or (b) an operation to copy at least a portion of the data of a storage object from a first storage device configured to remain powered on for continuous access, to a second storage device configured to remain in a low power state by default.

11. The system as recited in claim 7,

wherein the storage objects of the plurality of lifecycle transition execution iterations are determined by a transition candidate discovery iteration;

and,

wherein said perform the plurality of transition execution iterations is performed asynchronously with respect to the transition candidate discovery iteration.

12. The system as recited in claim 7, wherein the one or more respective hardware processors and memory are configured to:

subsequent to a determination that the storage operations for the particular transition job object have been performed, delete the particular transition job object from the storage service.

13. The system as recited in claim 7, wherein the one or more resources comprise a worker thread configured to:

subsequent to completion of a successful operation, send a task-done notification to the dispatcher, and

for an unsuccessful operation, send a task-failed notification to the dispatcher.

14. A method, comprising:

performing, by the storage lifecycle task dispatcher implemented via one or more hardware computing devices comprising one or more respective hardware processors and memory, a plurality of lifecycle transition execution iterations for storage objects of a multi-tenant storage service, wherein each lifecycle transition execution iteration of the plurality of lifecycle transition execution iterations is initiated based on one or more scheduling criteria;

identifying, by the storage lifecycle task dispatcher, during a particular lifecycle transition execution iteration of the plurality of lifecycle transition execution iterations, a set of transition job objects to be validated for execution;

validating a particular transition job object of the set; and

in response to said validating the particular transition job object of the set;

assigning one or more resources to initiate operations to implement lifecycle transitions indicated in the particular transition job object; and

monitoring a completion state of an operation to which a particular resource of the one or more resources is assigned.

15. The method as recited in claim 14, wherein a scheduling criterion of the one or more scheduling criteria is based at least in part on one or more of:

(a) a predetermined schedule, (b) a determination that a number of transition job objects stored in the storage service exceeds a threshold value, (c) one or more resource utilization metrics associated with the storage service, or (e) an indication of a status of a transition candidate discovery iteration performed by a transition job generator.

16. The method as recited in claim 14, wherein said monitoring the completion state of the operation to which the particular resource of the one or more resources is assigned comprises:

sending a task-done notification to the dispatcher subsequent to a successful operation.

17. The method as recited in claim 14, wherein a particular operation initiated to implement a lifecycle transition indicated in the particular transition job object comprises one or more of:

(a) an operation to repartition a storage object according to parameters of an erasure encoding policy or (b) an operation to copy at least a portion of the data of a storage object from a first storage device configured to remain powered on for continuous access, to a second storage device configured to remain in a low power state by default.

18. The method as recited in claim 14, further comprising:

prior to said performing, by the storage lifecycle task dispatcher, the plurality of lifecycle transition execution iterations for storage objects of the multi-tenant storage service,

determining the storage objects of the plurality of lifecycle transition execution iterations during a candidate discovery iteration,

wherein said performing the plurality of transition execution iterations is performed asynchronously with respect to the transition candidate discovery iteration.

19. The method as recited in claim 14, further comprising:

deleting, after determining that the storage operations for the particular transition job object have been performed, the particular transition job object from the storage service.

20. The method as recited in claim 14, wherein the one or more resources comprise a worker thread;

the method further comprising;

receiving, from the worker thread, a task-done notification for a successful operation.