Lifecycle management of storage clusters

• US 10,656,863 B1
• Filed: 11/03/2016
• Issued: 05/19/2020
• Est. Priority Date: 11/03/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Vines, Forrest MacKenzie, Nagasundaram, Avinash, Rinaldi, Andrew, Scott, Nathan David, Tian, Xingbang, Tseng, Jui Te
• Primary Examiner(s): Simonetti, Nicholas J
• Application Number:
  US15/343,083
• Time in Patent Office: 1,293 Days
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0616 in relation to life time, e.g. increasing Mean Time Between Failures [MTBF]
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Methods, systems, and computer-readable media for lifecycle management of multiple storage clusters are disclosed. A first set of data is loaded into a first storage cluster. The first storage cluster is selected for loading the first set of data based at least in part on the first set of data and the first storage cluster being associated with a first period of time. The first storage cluster is placed into a read-only state. A second set of data is loaded into a second storage cluster. The second storage cluster is selected for loading the second set of data based at least in part on the second set of data and the second storage cluster being associated with a second period of time.

Claims

1. A system, comprising:

one or more computing devices configured to implement a cluster lifecycle management system, wherein the cluster lifecycle management system is configured to;

launch a first storage cluster based at least in part on a schedule, the first storage cluster comprising two or more storage instances to store data from a data source and process queries directed to the data;

load a first set of data from the data source into the first storage cluster, wherein the first set of data and the first storage cluster are associated with a first period of time, and wherein the first storage cluster is selected for loading the first set of data based at least in part on the first set of data and the first storage cluster being associated with the first period of time;

launch a second storage cluster based at least in part on the schedule, the second storage cluster comprising two or more storage instances to store data from the data source and process queries directed to the data;

place the first storage cluster into a read-only state after the first set of data is loaded into the first storage cluster; and

load a second set of data from the data source into the second storage cluster, wherein the second set of data and the second storage cluster are associated with a second period of time, and wherein the second storage cluster is selected for loading the second set of data based at least in part on the second set of data and the second storage cluster being associated with the second period of time.

2. The system as recited in claim 1, wherein the cluster lifecycle management system is further configured to:

receive a query indicating a time range;

select the first storage cluster or the second storage cluster for the query based at least in part on matching the time range indicated in the query to the first period of time associated with the first storage cluster or the second period of time associated with the second storage cluster; and

retrieve, from the first storage cluster or the second storage cluster selected for the query, one or more elements of data responsive to the query.

3. The system as recited in claim 1, wherein the cluster lifecycle management system is further configured to:

based at least in part automated enforcement of a data retention policy, terminate the first storage cluster in the read-only state and store a snapshot of the first storage cluster.

4. The system as recited in claim 1, wherein the second storage cluster is launched with a different storage capacity than the first storage cluster based at least in part on a different anticipated storage requirement for the second period of time than for the first period of time.

5. A computer-implemented method, comprising:

performing, by one or more computing devices that implement a storage lifecycle management system;

loading a first set of data from a data source into a first storage resource that comprises two or more storage instances, the first storage resource to process queries directed to the first set of data, wherein the first set of data and the first storage resource are associated with a first period of time, and wherein the first storage resource is selected for loading the first set of data based at least in part on the first set of data and the first storage resource being associated with the first period of time;

placing the first storage resource into a read-only state after the first set of data is loaded into the first storage resource; and

loading a second set of data from the data sources into a second storage resource that comprises two or more storage instances, the second storage resource to process queries directed to the second set of data, wherein the second set of data and the second storage resource are associated with a second period of time, and wherein the second storage resource is selected for loading the second set of data based at least in part on the second set of data and the second storage resource being associated with the second period of time.

6. The method as recited in claim 5, further comprising:

receiving a query indicating a time range;

selecting the first storage resource or the second storage resource for the query based at least in part on matching the time range indicated in the query to the first period of time associated with the first storage resource or the second period of time associated with the second storage resource; and

retrieving, from the first storage resource or the second storage resource selected for the query, one or more elements of data responsive to the query.

7. The method as recited in claim 5, further comprising:

performing storage optimization on the first storage resource after the first set of data is loaded into the first storage resource.

8. The method as recited in claim 5, further comprising:

based at least in part automated enforcement of a data retention policy, terminating the first storage resource in the read-only state and storing a snapshot of the first storage resource.

9. The method as recited in claim 8, further comprising:

restoring the snapshot of the first storage resource to a newly launched storage resource.

10. The method as recited in claim 8, further comprising:

based at least in part automated enforcement of a snapshot retention policy, deleting the snapshot of the first storage resource.

11. The method as recited in claim 5, wherein the second storage resource is launched with a different storage capacity than the first storage resource based at least in part on a different anticipated storage requirement for the second period of time than for the first period of time.

12. The method as recited in claim 5, further comprising:

launching the first storage resource using an automated scheduler based at least in part on the first period of time; and

launching the second storage resource using the automated scheduler based at least in part on the second period of time.

13. The method as recited in claim 5, wherein the first period of time overlaps with the second period of time.

14. The method as recited in claim 5, wherein at least a portion of the second set of data is loaded into the second storage resource before the first storage resource is placed into the read-only state.

15. A computer-readable storage medium storing program instructions computer-executable to perform:

loading a first set of data into a first storage cluster that comprises two or more storage instances, the first storage cluster to process queries directed to the first set of data, wherein the first set of data and the first storage cluster are associated with a first period of time, and wherein the first storage cluster is selected for loading the first set of data based at least in part on the first set of data and the first storage cluster being associated with the first period of time;

placing the first storage cluster into a read-only state after the first period of time has elapsed; and

loading a second set of data into a second storage cluster that comprises two or more storage instances, the second storage cluster to process queries directed to the second set of data, wherein the second set of data and the second storage cluster are associated with a second period of time, and wherein the second storage cluster is selected for loading the second set of data based at least in part on the second set of data and the second storage cluster being associated with the second period of time.

16. The computer-readable storage medium as recited in claim 15, wherein the program instructions are further computer-executable to perform:

receiving a query indicating a time range;

selecting the first storage cluster or the second storage cluster for the query based at least in part on matching the time range indicated in the query to the first period of time associated with the first storage cluster or the second period of time associated with the second storage cluster; and

retrieving, from the first storage cluster or the second storage cluster selected for the query, one or more elements of data responsive to the query.

17. The computer-readable storage medium as recited in claim 16, wherein the first storage cluster is selected for the query based at least in part on the first storage cluster being in the read-only state and the second storage cluster not being in the read-only state.

18. The computer-readable storage medium as recited in claim 15, wherein the program instructions are further computer-executable to perform:

based at least in part automated enforcement of a data retention policy, terminating the first storage cluster in the read-only state and storing a snapshot of the first storage cluster.

19. The computer-readable storage medium as recited in claim 15, wherein the second storage cluster is launched with a different storage capacity than the first storage cluster based at least in part on a different anticipated storage requirement for the second period of time than for the first period of time.

20. The computer-readable storage medium as recited in claim 15, wherein the program instructions are further computer-executable to perform:

launching the first storage cluster using an automated scheduler based at least in part on arrival of or anticipation of the first period of time; and

launching the second storage cluster using the automated scheduler based at least in part on arrival of or anticipation of the second period of time.