Monitoring and automatic scaling of data volumes

• US 9,207,984 B2
• Filed: 03/31/2009
• Issued: 12/08/2015
• Est. Priority Date: 03/31/2009
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Sivasubramanian, Swaminathan, McAlister, Grant Alexander MacDonald, Milovanovic, Milovan
• Primary Examiner(s): SISON, JUNE Y
• Application Number:
  US12/415,998
• Publication Number:
  US 20100250748A1
• Time in Patent Office: 2,443 Days
• Field of Search: 709/226
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3442 for planning or managing the needed capacity
• CPC Class Codes: G06F 21/6209 to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 2201/875 Monitoring of systems including the internet
• CPC Class Codes: G06F 2201/88 Monitoring involving counting
• CPC Class Codes: G06F 9/5011 the resources being hardware resources other than CPUs, Servers and Terminals
• CPC Class Codes: G06F 9/5016 the resource being the memory

Abstract

Aspects of a data environment, such as various capacities of data stores and instances, can be managed using a separate control environment. A monitoring component of the control environment can periodically communicate with the data environment to obtain performance information. The information is analyzed, using algorithms such as trending and extrapolation algorithms, to determine any recommended scaling of resources in the data environment. The scaling can be performed automatically, or as authorized by a customer. A workflow can be instantiated that includes tasks necessary to perform the scaling. The scaling of storage capacity can be performed without affecting the availability of the data store.

Claims

1. A computer-implemented method of scaling storage capacity in a data environment using a separate control environment, comprising:

under control of one or more computer systems configured with executable instructions,

obtaining performance information for a logical data volume in the data environment, the performance information including at least one of storage usage information or storage capacity information for the logical data volume, the logical data volume corresponding to data stored among one or more physical volumes;

extracting historical information for the logical data volume, the historical information representing values of at least some of the performance information over a period of time;

analyzing the performance information and the historical information using at least one prediction algorithm to predict that an anticipated usage value will fall outside of at least one specified range within at least one specified time period;

selecting a scaling action for adjusting a storage capacity of the logical data volume for the anticipated usage value to fall inside the at least one specified range within the at least one specified time period based at least in part upon determining that a cost of implementing the scaling action is a lowest cost among a plurality of scaling options;

determining whether authorization is granted for implementing the scaling option based at least in part upon the cost of implementing the scaling option not exceeding a specified cost threshold; and

executing a workflow in the separate control environment for adjusting the storage capacity of the logical data volume in the data environment based at least in part upon the scaling option in response to determining that the authorization is granted, the storage capacity being adjusted in the data environment by;

(a) changing a number of the one or more physical volumes that collectively provide the storage capacity for the logical data volume without reducing availability of the logical data volume and (b) balancing input output performance among the one or more physical volumes.

2. The computer-implemented method of claim 1, wherein changing the number of the one or more physical volumes comprises:

initiating at least one new physical volume; and

extending the logical data volume by attaching the at least one new physical volume.

3. The computer-implemented method of claim 1, wherein balancing the input output performance among the one or more physical volumes is based at least in part on a number of available physical extents for the one or more physical volumes.

4. A computer-implemented method of scaling aspects of a data environment using a separate control environment, comprising:

under control of one or more computer systems configured with executable instructions,

obtaining performance information for at least one resource of a database in the data environment, the performance information including at least one of usage information or capacity information for the at least one resource, the at least one resource associated with a logical device corresponding to one or more physical devices;

extracting historical information for the at least one resource, the historical information representing values of at least some of the performance information over a period of time;

analyzing the performance information and the historical information using at least one prediction algorithm to predict that an anticipated usage value for the at least one resource will fall outside of at least one specified range within at least one specified time period;

selecting a scaling action, from among a plurality of scaling options, for adjusting a capacity of the at least one resource, the capacity of the at least one resource being adjusted based at least in part upon the anticipated usage value, the scaling action being selected based at least in part upon a respective cost corresponding to each of the plurality of scaling actions;

determining whether authorization is granted for the scaling option based at least in part upon the cost corresponding to the scaling option; and

executing a workflow in the separate control environment for adjusting the capacity of the at least one resource in the data environment based at least in part upon the scaling option in response to determining that the authorization is granted, the capacity being adjusted in the data environment by;

(a) changing a number of the one or more physical devices that collectively provide the capacity for the logical device and (b) balancing input output performance among the one or more physical devices.

5. The computer-implemented method of claim 4, wherein the at least one resource includes at least one of a processing component, a data storage component, a memory component, a communications component, a network I/O (input/output) component, or a data I/O component.

6. The computer-implemented method of claim 4, wherein executing the workflow comprises:

generating a series of tasks for adjusting the capacity; and

for each of the series of tasks, passing state information from a monitoring component in the control environment to a host manager in the data environment, wherein the host manager is operable to execute the task and return a response to the monitoring component.

7. The computer-implemented method of claim 6, further comprising:

determining a success or failure of each of the series of tasks before performing any subsequent task.

8. The computer-implemented method of claim 6, further comprising:

when a task is determined to have failed, retrying the task at least once and determining success or failure of each retry; and

when the task and the retry are determined to have failed, failing the workflow.

9. The computer-implemented method of claim 8, wherein:

failing the workflow includes rolling back each previously-executed task of the workflow.

10. The computer-implemented method of claim 4 further comprising:

when adjusting the capacity of the at least one resource is determined not to be authorized, contacting a user for the authorization before executing the workflow.

11. The computer-implemented method of claim 4 further comprising:

enabling a user to provide different levels of authorization for the at least one resource.

12. The computer-implemented method of claim 4, further comprising, after predicting the anticipated usage value for the at least one resource:

sending information including the anticipated usage value to an authorized user of the at least one resource; and

enabling the authorized user to call into the control environment to request the adjusting of the capacity of the at least one resource.

13. The computer-implemented method of claim 4, wherein the scaling action corresponds to a lowest cost among the plurality of scaling actions.

14. A system for scaling aspects of a data environment using a separate control environment, comprising:

at least one hardware processor; and

hardware memory including instructions that, when executed by the at least one processor, cause the system to;

obtain performance information for at least one resource of a database in the data environment, the performance information including at least one of usage information or capacity information for the at least one resource, the at least one resource including a logical device volume corresponding to one or more physical devices;

extract historical information for the at least one resource, the historical information representing values of at least some of the performance information over a period of time;

analyze the performance information and the historical information using at least one prediction algorithm to predict that an anticipated usage value for the at least one resource will fall outside of at least one specified range within at least one specified time period;

select a scaling action, from among a plurality of scaling options, for adjustment of a capacity of the at least one resource, the adjustment being based at least in part upon the anticipated usage value, selection of the scaling action being based at least in part upon a respective cost corresponding to each of the plurality of scaling actions;

determine whether authorization is granted for the scaling option based at least in part upon the cost corresponding to the scaling option; and

execute a workflow in the separate control environment for the adjustment of the capacity of the at least one resource in the data environment based at least in part upon the scaling option in response to a determination that the authorization is granted, the adjustment including (a) a changing of a number of the one or more physical devices that collectively provide the capacity for the logical device and (b) a balancing of input output performance among the one or more physical devices.

15. The system of claim 14, wherein:

the at least one resource includes at least one of a processing component, a data storage component, a memory component, a communications component, a network I/O component, or a data I/O component.

16. The system of claim 14, wherein the memory further includes instructions that, when executed by the processor, cause the system to:

when the determination is that the authorization is not granted, contact a user for the authorization before executing the workflow.

17. The system of claim 14, wherein the memory further includes instructions that, when executed by the processor, cause the system to:

enable a user to provide different levels of authorization for the at least one resource.

18. The system of claim 14, wherein the memory further includes instructions that, when executed by the processor, cause the system to:

send information including the anticipated usage value to an authorized user of the at least one resource; and

enable the authorized user to call into the control environment to request for the adjustment of the capacity of the at least one resource.

19. A computer program product embedded in a non-transitory computer-readable storage medium and including instructions that, when executed by at least one computing device, cause the at least one computing device to:

obtain performance information for at least one resource of a database in a data environment, the performance information including at least one of usage information or capacity information for the at least one resource, the at least one resource including a logical device corresponding to one or more physical devices;

extract historical information for the at least one resource, the historical information representing values of at least some of the performance information over a period of time;

analyze the performance information and the historical information using at least one prediction algorithm to predict that an anticipated usage value for the at least one resource will fall outside of at least one specified range within at least one specified time period;

select a scaling action, from among a plurality of scaling options, for adjustment of a capacity of the at least one resource, the adjustment being based at least in part upon the anticipated usage value, selection of the scaling action being based at least in part upon a respective cost corresponding to each of the plurality of scaling actions;

determine whether authorization is granted for the scaling option based at least in part upon the cost corresponding to the scaling option; and

execute a workflow in a control environment for the adjustment of the capacity of the at least one resource in the data environment based at least in part upon the scaling option in response to a determination that the authorization is granted, the adjustment including;

(a) a changing of a number of the one or more physical devices that collectively provide the capacity for the logical device and (b) a balancing of input output performance among the one or more physical devices.

20. The computer program product embedded in the non-transitory storage medium of claim 19, wherein:

the at least one resource includes at least one of a processing component, a data storage component, a memory component, a communications component, a network I/O component, or a data I/O component.

21. The computer program product embedded in the non-transitory storage medium of claim 19, further including instructions that, when executed by at least one computing device, cause the at least one computing device to:

when the determination is that the authorization is not granted, contact a user for the authorization before executing the workflow.

22. The computer program product embedded in the non-transitory storage medium of claim 19, further including instructions that, when executed by at least one computing device, cause the at least one computing device to:

enable a user to provide different levels of authorization for the at least one resource.

23. The computer program product embedded in the non-transitory storage medium of claim 19, further including instructions that, when executed by at least one computing device, cause the at least one computing device to:

send information including the anticipated usage value to an authorized user of the at least one resource; and

enable the authorized user to call into the control environment to request for the adjustment of the capacity of the at least one resource.