Monitoring and Automatic Scaling of Data Volumes

• US 20100250748A1
• Filed: 03/31/2009
• Published: 09/30/2010
• Est. Priority Date: 03/31/2009
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: MILOVANOVIC, Milovan, MCALISTER, Grant Alexander MacDonald, SIVASUBRAMANIAN, Swaminathan
• Granted Patent: US 9,207,984 B2
• Time in Patent Office: Days
• US Class Current: 709/226
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3442 for planning or managing the needed capacity
• CPC Class Codes: G06F 21/6209 to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 2201/875 Monitoring of systems including the internet
• CPC Class Codes: G06F 2201/88 Monitoring involving counting
• CPC Class Codes: G06F 9/5011 the resources being hardware resources other than CPUs, Servers and Terminals
• CPC Class Codes: G06F 9/5016 the resource being the memory

Abstract

Aspects of a data environment, such as various capacities of data stores and instances, can be managed using a separate control environment. A monitoring component of the control environment can periodically communicate with the data environment to obtain performance information. The information is analyzed, using algorithms such as trending and extrapolation algorithms, to determine any recommended scaling of resources in the data environment. The scaling can be performed automatically, or as authorized by a customer. A workflow can be instantiated that includes tasks necessary to perform the scaling. The scaling of storage capacity can be performed without affecting the availability of the data store.

Claims

1. A computer-implemented method of scaling storage capacity in a data environment using a separate control environment, comprising:

under control of one or more computer systems configured with executable instructions,

obtaining performance information for a logical data volume, the performance information including at least one of a storage usage and a storage capacity for the logical data volume;

extracting historical information for the logical data volume, the historical information representing values of at least some of the performance information over a period of time;

analyzing the performance information and the historical information using at least one prediction algorithm to predict an anticipated usage value at a determined point in the future; and

executing a workflow in the control environment for adjusting the storage capacity of the logical data volume in the data environment based at least in part upon the anticipated usage value,

wherein the storage capacity is adjusted in the data environment without affecting an availability of the logical data volume.

2. The computer-implemented method of claim 1, wherein adjusting the storage capacity for the logical volume comprises:

initiating at least one new data volume; and

extending the logical data volume by attaching the at least one new data volume.

3. The computer-implemented method of claim 2, further comprising:

rebalancing the logical data volume across each existing and new data volume of the logical data volume based at least in part on the number of available physical extents for each data volume.

4. A computer-implemented method of scaling aspects of a data environment using a separate control environment, comprising:

under control of one or more computer systems configured with executable instructions,

obtaining performance information for at least one resource in the data environment, the performance information including at least one of usage information and capacity information for the at least one resource;

extracting historical information for the at least one resource, the historical information representing values of at least some of the performance information over a period of time;

analyzing the performance information and the historical information using at least one prediction algorithm to predict an anticipated usage value for the at least one resource; and

executing a workflow in the control environment for adjusting the capacity of the at least one resource in the data environment based at least in part upon at least one of the usage information and the anticipated usage value, the workflow including at least one task configured to call into the data environment to perform the adjustment.

5. The computer-implemented method of claim 4, wherein the at least one resource includes at least one of a processing component, a data storage component, a memory component, a communications component, a network I/O (input/output) component, and a data I/O component.

6. The computer-implemented method of claim 4, wherein executing a workflow further comprises:

generating a series of tasks for adjusting the capacity; and

for each of the series of tasks, passing state information from a monitoring component in the control environment to a host manager in the data environment, wherein the host manager is operable to execute the task and return a response to the monitoring component.

7. The computer-implemented method of claim 6, further comprising:

determining a success or failure of each of the series of tasks before performing any subsequent task.

8. The computer-implemented method of claim 6, further comprising:

when a task is determined to have failed, retrying the task at least once and determining the success or failure of each retry; and

when the task and the at least one retry are determined to have failed, failing the entire workflow.

9. The computer-implemented method of claim 8, wherein:

failing the workflow includes rolling back each previously-executed task of the workflow.

10. The computer-implemented method of claim 6, further comprising:

determining an authorization status before executing the workflow.

11. The computer-implemented method of claim 10, further comprising:

when adjusting the capacity of the at least one resource is determined not to be authorized, contacting a user for authorization before executing the workflow.

12. The computer-implemented method of claim 10, further comprising:

enabling the user to provide different levels of authorization for each resource.

13. The computer-implemented method of claim 12, wherein:

the different levels of authorization include at least one authorization based at least in part upon a determined cost of the adjustment.

14. The computer-implemented method of claim 4, further comprising, after predicting the anticipated usage value for the at least one resource:

sending information including the anticipated usage value to an authorized user of the at least one resource; and

enabling the authorized user to call into the control environment to request the adjusting of the capacity of the at least one resource.

15. The computer-implemented method of claim 4, further comprising:

after predicting the anticipated usage value for the at least one resource, analyzing each of a set of scaling actions for the anticipated usage value and selecting the scaling action representing the lowest cost.

16. A system for scaling aspects of a data environment using a separate control environment, comprising:

at least one processor; and

memory including instructions that, when executed by the at least one processor, cause the system to;

obtain performance information for at least one resource in the data environment, the performance information including at least one of usage information and capacity information for the at least one resource;

extract historical information for the at least one resource, the historical information representing values of at least some of the performance information over a period of time;

analyze the performance information and the historical information using at least one prediction algorithm to predict an anticipated usage value for the at least one resource at a determined point in the future; and

execute a workflow in the control environment for adjusting the capacity of the at least one resource in the data environment based at least in part upon at least one of the usage information and the anticipated usage value, the workflow including at least one task configured to call into the data environment to perform the adjustment.

17. The system of claim 16, wherein:

the at least one resource includes at least one of a processing component, a data storage component, a memory component, a communications component, a network I/O component, and a data I/O component.

18. The system of claim 16, wherein the memory further includes instructions that, when executed by the processor, cause the system to:

determine an authorization status before executing the workflow; and

when adjusting the capacity of the at least one resource is determined not to be authorized, contact a user for authorization before executing the workflow.

19. The system of claim 16, wherein the memory further includes instructions that, when executed by the processor, cause the system to:

enable the user to provide different levels of authorization for each resource.

20. The system of claim 16, wherein the memory further includes instructions that, when executed by the processor, cause the system to:

send information including the anticipated usage value to an authorized user of the at least one resource; and

enable the authorized user to call into the control environment to request the adjusting of the capacity of the at least one resource.

21. A computer program product embedded in a computer-readable medium and including instructions that, when executed by at least one computing device, cause the at least one computing device to:

obtain performance information for at least one resource in the data environment, the performance information including at least one of usage information and capacity information for the at least one resource;

extract historical information for the at least one resource, the historical information representing values of at least some of the performance information over a period of time;

analyze the performance information and the historical information using at least one prediction algorithm to predict an anticipated usage value for the at least one resource at a determined point in the future; and

execute a workflow in the control environment for adjusting the capacity of the at least one resource in the data environment based at least in part upon at least one of the usage information and the anticipated usage value, the workflow including at least one task configured to call into the data environment to perform the adjustment.

22. The computer program product of claim 21, wherein:

the at least one resource includes at least one of a processing component, a data storage component, a memory component, a communications component, a network I/O component, and a data I/O component.

23. A computer program product according to claim 21, further including instructions that, when executed by at least one computing device, cause the at least one computing device to:

determine an authorization status before executing the workflow; and

when adjusting the capacity of the at least one resource is determined not to be authorized, contact a user for authorization before executing the workflow.

24. A computer program product according to claim 21, further including instructions that, when executed by at least one computing device, cause the at least one computing device to:

enable the user to provide different levels of authorization for each resource.

25. A computer program product according to claim 21, further including instructions that, when executed by at least one computing device, cause the at least one computing device to:

send information including the anticipated usage value to an authorized user of the at least one resource; and

enable the authorized user to call into the control environment to request the adjusting of the capacity of the at least one resource.