Cloning and recovery of data volumes

• US 8,332,365 B2
• Filed: 03/31/2009
• Issued: 12/11/2012
• Est. Priority Date: 03/31/2009
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Milovanovic, Milovan, McAlister, Grant A. M.
• Primary Examiner(s): Dao, Thuy
• Assistant Examiner(s): Lu, Hua
• Application Number:
  US12/415,968
• Publication Number:
  US 20100250499A1
• Time in Patent Office: 1,351 Days
• Field of Search: None
• US Class Current: 707/674
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 2201/875 Monitoring of systems including the internet

Abstract

Aspects of a data environment, such as the monitoring, backup, and recovery of data stores and instances, are managed using a separate control environment. A monitoring component of the control environment can periodically communicate with the data environment to determine any necessary actions to be performed, such as to recover from faults or events for a data instance in the data environment. A workflow can be instantiated that includes tasks necessary to perform a recovery action. Tasks of the workflow can instantiate a new data instance and, using an identifier for the original data instance, access a snapshot for the data instance and copy the data to the new instance. Log files also can be accessed and applied to the new data instance to cause the new instance to represent the state of the original data instance at a particular point in time.

Claims

1. A computer-implemented method of recovering from a failure of a data instance in a data environment using a separate control environment, comprising:

under control of one or more computer systems configured with executable instructions,

sending a status request from a monitoring component in a control environment to a host manager in a data environment, the host manager operable to monitor a status of a source data store in the data environment, the source data store containing a plurality of data entries;

analyzing, in the control environment, a response received from the host manager to determine whether a problem state exists with the source data store that requires a recovery action;

when the problem state is determined to exist, causing one or more processors of the one or more computer systems to determine a recovery workflow based at least in part upon a type of problem and a time at which the problem occurred in the source data store; and

executing the recovery workflow in the control environment, the recovery workflow receiving an identifier of the source data store and causing the source data store to be restored by (a) provisioning a new data store in the data environment for the identifier of the source data store, and (b) copying a snapshot associated with the identifier of the source data store into the new data store, the snapshot being based at least in part on the time at which the problem occurred in the source data store.

2. The computer-implemented method of claim 1, wherein:

executing the recovery workflow in the control environment further causes log data to be applied to the new data store in order to update a state of the new data store.

3. The computer-implemented method of claim 2, wherein:

the state of the new data store is updated to a most recent time for which the log data exists.

4. The computer-implemented method of claim 2, wherein:

the state of the new data store is updated to a time based at least in part upon the response received from the host manager.

5. The computer-implemented method of claim 2, wherein:

the log data is selected from at least one of a log snapshot and log information stored in the data environment.

6. The computer-implemented method of claim 2, wherein:

applying the log data includes iteratively applying more recent versions of the log data in order to update the state of the new data store.

7. The computer-implemented method of claim 1, further comprising:

periodically sending a status request from the monitoring component in the control environment to the host manager in the data environment; and

analyzing each response received from the host manager,

wherein the status of the source data store is automatically monitored over time.

8. The computer-implemented method of claim 1, further comprising:

storing snapshots for the new data store at a determined interval.

9. The computer-implemented method of claim 1, wherein:

snapshots are stored to the data environment using a first interval, and stored to the control environment at a second interval.

10. The computer-implemented method of claim 1, further comprising:

storing information for the problem to a job queue in the data environment; and

detecting the information in the job queue before determining the recovery workflow.

11. The computer-implemented method of claim 1, wherein:

at least one aspect of the new data store is different from the source data store.

12. A computer-implemented method of cloning a data instance in a data environment using a separate control environment, comprising:

under control of one or more computer systems configured with executable instructions,

receiving, to a control environment, a request to generate a second data instance corresponding to a state of a first data instance;

selecting a time associated with the state of the first data instance;

in response to the request, causing one or more processors to determine a workflow for generating the second data instance based at least in part upon a type of the request and the selected time of the state of the first data instance; and

executing the determined workflow in the control environment, the determined workflow receiving an identifier of the first data instance and causing the first data instance to be cloned by (a) provisioning the second data instance in a data environment for the identifier of the first data instance, and (b) copying a snapshot associated with the identifier of the first data instance into the second data instance the snapshot being based at least in part on the selected time associated with the state of the first data instance.

13. The computer-implemented method of claim 12, wherein:

executing the determined workflow in the control environment further causes log data to be applied to the second data instance in order to update a second state of the second data instance to the state of the first data instance at the selected time.

14. The computer-implemented method of claim 13, wherein:

the second state of the second data instance is updated to a second time determined by the request.

15. The computer-implemented method of claim 12, wherein:

the second data instance is instantiated, populated, and updated without affecting an availability of the first data instance.

16. The computer-implemented method of claim 12, further comprising:

restricting access to the first data instance during execution of the determined workflow, wherein the second data instance will be an exact data replica of the first data instance.

17. The computer-implemented method of claim 12, wherein copying the snapshot associated with the identifier of the first data instance includes:

determining a logical volume group corresponding to the first data instance, the logical volume group including at least one data volume; and

copying data from a backup for each data volume of the logical volume group.

18. The computer-implemented method of claim 12, wherein:

a user is able to use the second data instance to perform an operation including at least one of testing an application requiring data access, storing a state of the first data instance at a point in time, testing a data storage configuration, or modifying a data schema before applying the data schema to the first data instance.

19. The computer-implemented method of claim 12, further comprising:

copying data from the snapshot to multiple data instances, each of the multiple data instances corresponding to a new partitioning of a data store; and

deleting copied data from each of the multiple data instances not to be included in a respective partitioning.

20. A computer-implemented method of hibernating a data instance in a data environment using a separate control environment, comprising:

under control of one or more computer systems configured with executable instructions,

receiving, to a control environment, a first request to hibernate a first data instance;

in response to the first request, storing a snapshot of a current state of the first data instance;

the snapshot of the current state being based at least in part on a time that the request to hibernate the first data instance was received, and releasing the first data instance in a data environment;

receiving, to the control environment, a second request to resume the first data instance;

causing one or more processors to determine a workflow for generating a second data instance corresponding to the first data instance; and

executing the determined workflow in the control environment, the determined workflow receiving an identifier of the first data instance and causing the first data instance to be resumed by (a) provisioning the second data instance in the data environment for the identifier of the first data instance, and (b) copying the snapshot associated with the identifier of the first data instance into the second data instance, the snapshot being based at least in part on the time that the first request to hibernate the first data instance was received.

21. A system for controlling a data environment using a separate control environment, comprising:

at least one processor; and

memory including instructions that, when executed by the at least one processor, cause the at least one processor to perform;

sending a request from a monitoring component in the control environment to a host manager in a data environment, the host manager operable to monitor a status of a source data store in the data environment, the source data store containing a plurality of data entries;

analyzing, in the control environment, a response received from the host manager to determine whether to copy a state of the source data store;

determining a need to copy a state of a source data store based at least in part on the response received from the host manager;

selecting a time associated with the state of the source data store;

causing one or more processors to determine a workflow based at least in part upon a type of the need and the selected time associated with the state of the source data store; and

executing the determined workflow in the control environment, the determined workflow receiving an identifier of the source data store and causing the source data store to be copied by (a) provisioning a new data store in the data environment for the identifier of the source data store, and (b) copying a snapshot associated with the identifier of the source data store into the new data store, the snapshot being based at least in part on the selected time associated with the state of the source data store.

22. The system of claim 21, wherein:

the request is a status request;

analyzing, in the control environment, the response received from the host manager comprises analyzing the response to determine whether a problem exists with the source data store that requires a recovery action; and

the need to copy is based on the response.

23. The system of claim 21, wherein the memory further includes instructions that, when executed by the at least one processor, cause the system to:

receive, to the control environment, a request to clone the source data store, the request determining the need to copy.

24. A computer program product embedded in a non-transitory computer-readable medium and including instructions that, when executed by at least one computing device, causes the following to be performed:

send a request from a monitoring component in a control environment to a host manager in a data environment, the host manager operable to monitor a status of a source data store in the data environment, the source data store containing a plurality of data entries;

analyze, in the control environment, a response received from the host manager to determine whether to copy a state of the source data store;

determine a need to copy the state of the source data store based at least in part on the response received from the host manager;

select a time associated with the state of the source data store;

cause one or more processors to determine a workflow based at least in part upon a type of need and the selected time associated with the state of the source data store; and

execute the determined workflow in the control environment, the determined workflow receiving an identifier of the source data store and causing the source data store to be copied by (a) provisioning a new data store in the data environment for the identifier of the source data store, and (b) copying a snapshot associated with the identifier of the source data store into the new data store, the snapshot being based at least in part on the selected time associated with the state of the source data store.

25. The computer program product according to claim 24, wherein:

the request is a status request;

analyze, in the control environment, the response received from the host manager comprises analyzing the response to determine whether a problem exists with the source data store that requires a recovery action; and

the need to copy is based on the response.

26. A computer program product according to claim 24, further including instructions that, when executed by at least one computing device, cause the at least one computing device to:

receive, to the control environment, a request to clone the source data store, the request determining the need to copy.