PROVIDING A RELIABLE BACKING STORE FOR BLOCK DATA STORAGE

• US 20100036931A1
• Filed: 08/08/2008
• Published: 02/11/2010
• Est. Priority Date: 08/08/2008
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: DeSantis, Peter N., Certain, Tate Andrew, Garman, Matthew S., Paterson-Jones, Roland, Jorgensen, Atle Normann
• Granted Patent: US 7,831,682 B2
• Time in Patent Office: Days
• US Class Current: 709/214
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2071 using a plurality of controllers
• CPC Class Codes: G06F 11/2082 Data synchronisation
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

Techniques are described for managing access of executing programs to non-local block data storage. In some situations, a block data storage service uses multiple server storage systems to reliably store copies of network-accessible block data storage volumes that may be used by programs executing on other physical computing systems, and at least some stored data for some volumes may also be stored on remote archival storage systems. A group of multiple server block data storage systems that store block data volumes may in some situations be co-located at a data center, and programs that use volumes stored there may execute on other computing systems at that data center, while the archival storage systems may be located outside the data center. The data stored on the archival storage systems may be used in various ways, including to reduce the amount of data stored in at least some volume copies.

Claims

1. A method for a computing system of a block data storage service to maintain access of executing programs to remote stored block data, the method comprising:

receiving a request to initiate access of a first executing program to non-local block data storage provided by a block data storage service, the first program executing on a first of a plurality of computing systems that are co-located at a first geographical location and that share one or more internal networks, the block data storage service using a first group of multiple other of the plurality of computing systems as block data storage systems that provide block data storage to multiple executing programs, the first computing system not being part of the first group; and

under control of a system manager module of the block data storage service, managing the providing of the block data storage to the multiple executing programs, by;

after receiving indications of one or more data access requests initiated by the first executing program to the non-local block data storage provided by the block data storage service, automatically responding by performing the data access requests on a first block data storage volume that has a primary copy stored on a second computing system and that has a mirror copy stored on a third computing system, the second and third computing systems each being part of the first group of block data storage systems, the performing of the data access requests causing one or more modifications to block data stored on the first block data storage volume in such a manner that the primary and mirror copies of the first block data storage volume each store the same block data;

at a first time after the performing of the data access requests, creating a first snapshot copy of the first block data storage volume on one or more remote archival data storage systems at a second geographical location that is distinct from the first geographical location, the first snapshot copy including a copy of the block data stored on the first block data storage volume at the first time; and

after creating the first snapshot copy and receiving indications of one or more other data access requests initiated by the first executing program to the provided non-local block data storage,

automatically responding by performing the other data access requests on the primary copy stored on the second computing system, the performing of a subset of the other data access requests causing one or more further modifications to the block data stored on the primary copy;

maintaining the mirror copy by modifying the block data stored on the third computing system to include only the further modified block data from the performing of the other data access requests subset after the creating of the first snapshot copy, such that the maintained mirror copy on the third computing system lacks a subset of the block data stored on the second computing system for the primary copy of the block data storage volume, the stored block data subset corresponding to the stored block data that has not been modified since the creating of the first snapshot copy; and

after determining to promote the mirror copy of the first block data storage volume on the third computing system to be a new primary copy of the first block data storage volume, automatically restoring the mirror copy on the third computing system to include the subset of the stored block data by retrieving the subset of the stored block data from the first snapshot copy on the one or more remote archival data storage systems, such that the restored mirror copy on the third computing system matches the primary copy of the first block data storage volume before the promoting of the mirror copy.

2. The method of claim 1 further comprising, under control of a node manager module of the block data storage service that manages access of the first computing system to the internal networks:

receiving the data access requests and the other data access requests initiated by the first executing program, the data access requests and other data access requests initiated by interactions of the first executing program with a logical local storage device of the first computing system that represents the first block data storage volume; and

automatically responding to the received data access requests and to the received other data access requests by interacting over the internal networks with the system manager module to provide the indications of the data access requests and of the other data access requests so as to cause performance of the data access requests and the other data access requests on the first block data storage volume on behalf of the first executing program, the interacting over the internal networks being performed in a manner that is transparent to the first executing program.

3. The method of claim 2 wherein the multiple programs are executed by a program execution service on a second group of multiple of the plurality of computing systems, wherein the computing systems of the second group are distinct from the computing systems of the first group, wherein the first computing system hosts multiple virtual machines that are each able to execute at least one program, wherein the first executing program is one of the multiple programs and is a virtual machine image being executed by at least one of the multiple virtual machines hosted by the first computing system, and wherein the first node manager module executes as part of a virtual machine monitor for the first computing system.

4. A computer-implemented method for maintaining access of executing programs to stored block data, the method comprising:

receiving one or more indications of one or more data access requests initiated by a first program executing on a first computing system to access block data stored on a non-local block data storage volume, the block data storage volume provided by a distinct second data storage system that is at a common first geographical location with the first computing system and that is separated from the first computing system by one or more networks;

automatically responding to the received indications of the data access requests by interacting with the second data storage system on behalf of the first executing program so as to initiate performance of the data access requests on the block data storage volume provided by the second data storage system, the performance of the data access requests causing one or more modifications to the block data stored on the block data storage volume;

automatically maintaining a mirror copy of the block data storage volume on a third data storage system at the first geographical location, the third data storage system being distinct from the first computing system and second data storage system, the maintained mirror copy on the third data storage system lacking a subset of the block data stored on the second data storage system for the block data storage volume, the subset of the stored block data corresponding to one or more portions of the stored block data that have not been modified since a prior creation of a snapshot copy of the block data storage volume on one or more archival data storage systems at a second geographical location that is distinct from the first geographical location; and

after determining to make access available to the mirror copy of the block data storage volume on the third data storage system, automatically restoring the mirror copy on the third data storage system to include the subset of the stored block data by retrieving the subset of the stored block data from the snapshot copy on the one or more archival data storage systems.

5. The method of claim 4 wherein, before the determining to make the access available to the mirror copy of the block data storage volume on the third data storage system, the second data storage system provides an initial primary copy of the block data storage volume, and wherein the method further comprises, after the automatic restoring of the mirror copy, using the restored mirror copy on the third data storage system as a new primary copy of the block data storage volume, the using of the restored mirror copy including performing one or more additional indicated data access requests on the new primary copy of the block data storage volume provided by the third data storage system.

6. The method of claim 5 wherein the determining to make the access available to the mirror copy of the block data storage volume on the third data storage system is based at least in part on the initial primary copy of the block data storage volume on the second data storage system becoming unavailable, the unavailability being based on at least one of a failure of the second data storage system, of a failure in connectivity to the second data storage system, and of an inability of the second data storage system to reliably access the stored block data storage volume, and wherein the restoring of the mirror copy is performed such that the restored mirror copy on the third data storage system matches the initial primary copy of the block data storage volume previously provided by the second data storage system.

7. The method of claim 4 further comprising, before the receiving of the indications of the data access requests initiated by the first program:

performing a first group of modifications to the block data stored on the block data storage volume on the second data storage system based on other previous data access requests initiated by the first program; and

creating the snapshot copy of the block data storage volume on the archival data storage systems at the second geographical location, the created snapshot copy including a copy of the block data stored on the block data storage volume on the second data storage system at a first time after the modifications of the first group are performed, such that the snapshot copy corresponds to the block data storage volume at the first time; and

wherein the stored block data subset is at least some of the stored block data in the snapshot copy.

8. The method of claim 7 wherein the snapshot copy created at the first time is a first snapshot copy, and wherein the method further comprises, after the performance of the one or more data access requests that cause the one or more modifications to the block data stored on the block data storage volume, creating a second snapshot copy of the block data storage volume on one or more archival data storage systems at a later second time, such that the second snapshot copy corresponds to the block data storage volume at the second time, the creating of the second snapshot copy including storing a copy of the modified block data from the performance of the one or more data access requests in a manner separate from the first snapshot copy and including sharing at least some of the block data stored in the first snapshot copy that has not been modified since the first time.

9. The method of claim 7 wherein the snapshot copy created at the first time is a first snapshot copy, and wherein the method further comprises:

at a second time earlier than the first time and before the performing of the first group of modifications, creating a second snapshot copy of the block data storage volume on one or more archival data storage systems at the second earlier time, such that the second snapshot copy corresponds to the block data storage volume at the second time before the performing of the first group of modifications;

after the performing of the first group of modifications and before the receiving of the indications of the data access requests initiated by the first program, creating the mirror copy of the block data storage volume on the third data storage system to include the block data stored on the second data storage system that is modified by the modifications of the first group, the created mirror copy not including at least some block data previously stored in the second snapshot copy that is not modified by the modifications of the first group; and

after the creating of the first snapshot copy and before the receiving of the indications of the data access requests initiated by the first program, updating the created mirror copy to remove the block data stored on the second data storage system that is previously modified by the modifications of the first group, the removing based at least in part on the first snapshot copy including a copy of the block data previously modified by the modifications of the first group.

10. The method of claim 4 further comprising, before the receiving of the indications of the one or more data access requests, attaching the block data storage volume to the first computing system for use by the first executing program, the attaching of the block data storage volume including associating a logical local block storage device of the first computing system with the block data storage volume provided by the second data storage system and being performed by a node manager module that manages access of the first computing system to the one or more networks, the node manager module being part of a block data storage service that previously creates the block data storage volume in response to a request from a user who is associated with the first executing program, and wherein the initiating by the first executing program of the one or more data access requests includes interactions by the first executing program with the logical local block storage device.

11. The method of claim 4 wherein the first computing system and the second and third data storage systems are a subset of a plurality of computing systems co-located at the first geographical location, wherein the plurality of computing systems includes multiple block data storage systems that are provided by a block data storage service, wherein the second and third data storage systems are each a distinct one of the multiple block data storage systems, wherein the first geographical location is a data center, wherein the second geographical location is external to the data center, and wherein the one or more archival data storage systems at the second geographical location are provided by a remote storage service.

12. The method of claim 11 wherein the first program is executed by a program execution service that executes multiple programs for multiple users on multiple of the plurality of computing systems, wherein the first computing system hosts multiple virtual machines that are each able to execute at least one program, wherein the first executing program is one of the multiple programs and is a virtual machine image being executed by at least one of the multiple virtual machines hosted by the first computing system, wherein the receiving of the indications of the first group of the data access requests and the receiving of the indications of the second group of the other data access requests and the automatic responding to the received indications of the first and second groups of data access requests are performed as part of execution of a virtual machine monitor for the first computing system, and wherein the first computing system includes one or more other actual local storage devices that are available for use by the first executing program.

13. A computer-readable medium whose contents enable one or more computing systems to maintain access of executing programs to stored block data, by performing a method comprising:

providing access of a first program executing on a first computing system to a block data storage volume provided by a second block data storage system, the provided access enabling the first program to initiate data access requests for block data stored on the block data storage volume;

automatically responding to one or more received indications of one or more data access requests initiated by the first program for the block data storage volume, the responding including initiating performance of the data access requests on the block data storage volume on the second block data storage system and including initiating performance of at least some of the data access requests on a distinct mirror copy of the block data storage volume on a third block data storage system, the indications received after a copy of the block data that is stored on the block data storage volume at an earlier time is created on one or more other remote archival storage systems, the mirror copy of the block data storage volume on the third block data storage system not including at least some block data that is currently stored in the block data storage volume on the second block data storage system and not including at least some of the block data that is stored in the copy of the block data storage volume on the remote archival storage systems; and

after the performance of the at least some data access requests on the mirror copy of the block data storage volume on the third block data storage system, updating the mirror copy on the third block data storage system to include the at least some block data that is currently stored in the block data storage volume on the second block data storage system, the updating including retrieving at least a portion of the at least some block data from the block data that is stored in the copy on the remote archival storage systems.

14. The computer-readable medium of claim 13 wherein the first computing system and the second and third block data storage systems are located at a single geographical location and separated by one or more networks, wherein the performance of the data access requests on the block data storage volume on the second block data storage system, the performance of the at least some data access requests on the mirror copy on the third block data storage system, and the updating of the mirror copy on the third block data storage system are automatically performed by a system manager module of a block data storage service, wherein the providing of the access of the first executing program to the block data storage volume is performed by a node manager module of the block data storage service that is associated with the first computing system to manage access of the first program to the one or more networks and includes providing a logical block data storage device that is local to the first computing system and that represents the block data storage volume provided by the second block data storage system, wherein the indications of the data access requests initiated by the first program are received by the node manager module after interactions by the first program with the logical local block data storage device, and wherein the automatic responding to the received indications of the data access requests is performed in part under control of the node manager module by initiating sending of the received data access requests over the one or more networks to the second block data storage system.

15. The computer-readable medium of claim 13 wherein the block data storage volume copy on the remote archival storage systems is a snapshot copy of the block data stored on the block data storage volume at the earlier time, wherein the first computing system and the second block data storage system and the third block data storage system are at a first geographical location and are separated by one or more networks, wherein the one or more remote archival storage systems are at a second geographical location distinct from the first geographical location, wherein at least one of the data access requests initiated by the first program cause one or more modifications to the block data stored on the block data storage volume on the second block data storage system, and wherein the updating of the mirror copy on the third block data storage system is performed in response to a determination to make access available to the mirror copy.

16. The computer-readable medium of claim 13 wherein the block data storage volume copy on the remote archival storage systems is at least one of a backup copy of the block data storage volume and a distinct second mirror copy of the block data storage volume, and wherein the automatic responding to the received indications of the data access requests initiated by the first program for the block data storage volume further includes updating the block data storage volume copy on the remote archival storage systems after the earlier time to reflect modifications from performance of at least some of the data access requests.

17. The computer-readable medium of claim 13 wherein the block data storage volume copy on the remote archival storage systems is a first snapshot copy of the block data stored on the block data storage volume at the earlier time, wherein at least one of the data access requests initiated by the first program cause one or more modifications to the block data stored on the block data storage volume on the second block data storage system, and wherein the method further comprises, after the automatic responding to the received indications of the data access requests initiated by the first program for the block data storage volume, creating on at least one of the remote archival storage systems a second snapshot copy of the block data that is currently stored on the block data storage volume, the creating of the second snapshot copy including storing a copy of the modified block data in a manner separate from the first snapshot copy and including sharing at least some of the block data stored in the first snapshot copy that has not been modified since the earlier time.

18. The computer-readable medium of claim 13 wherein the computer-readable medium is at least one of a memory of a computing system that stores the contents and a data transmission medium that includes a generated stored data signal containing the contents.

19. The computer-readable medium of claim 13 wherein the contents are instructions that when executed cause the one or more computing systems to perform the method.

20. A system configured to maintain access of executing programs to block data storage functionality, comprising:

one or more memories; and

a block data storage system manager module configured to provide a block data storage service that uses multiple block data storage systems to store block data storage volumes that are created by users of the block data storage service and accessed over one or more networks on behalf of one or more executing programs associated with the users, the providing of the block data storage service including;

creating one or more block data storage volumes for use by one or more executing programs, each of the block data storage volumes being stored on one of the multiple block data storage systems;

after access to a first of the created block data storage volumes that is stored on a first of the block data storage systems is provided to a first of the one or more executing programs, responding to one or more data access requests by the first program for the first block data storage volume by initiating performance of the one or more data access requests on the first block data storage volume stored on the first block data storage system, at least one of the one or more data access requests modifying block data stored on the first block data storage volume on the first block data storage system;

after the performance of the one or more data access requests on the first block data storage volume, creating on one or more other remote archival storage systems a copy of the block data stored on the first block data storage volume; and

after the creating of the copy on the remote archival storage systems,

responding to one or more other data access requests initiated by the first program for the first block data storage volume by initiating performance of the one or more other data access requests on the first block data storage volume stored on the first block data storage system, at least one of the one or more other data access requests further modifying the block data stored on the first block data storage volume on the first block data storage system; and

automatically providing a mirror copy of the first block data storage volume on a second of the block data storage systems, the providing of the mirror copy including storing on the second block data storage system a first subset of the stored block data in the first block data storage volume and including storing on the one or more other remote archival storage systems a second subset of the stored block data in the first block data storage volume.

21. The system of claim 20 wherein the copy on the remote archival storage systems of the stored block data is a snapshot copy of the block data stored in the first block data storage volume at a first time, wherein the first block data storage system and the second block data storage system and a computing system executing the first program are at a first geographical location and the one or more other remote archival storage systems are at a distinct second geographical location, wherein the first block data subset stored on the second block data storage system includes the stored block data that is further modified by the at least one other data access requests, wherein the second block data subset stored on the remote archival storage systems includes at least some of the block data stored in the copy created on the remote archival storage systems, and wherein the providing of the block data storage service further includes, at a second time after the providing of the mirror copy on the second block data storage system, automatically updating the mirror copy on the second block data storage system to include the second block data subset by retrieving the second block data subset from the snapshot copy on the remote archival data storage systems, the updating of the mirror copy being performed in response to a determination to make access available to the mirror copy on the second block data storage system.

22. The system of claim 20 wherein the creating of the first block data storage volume further includes creating a mirror copy of the first block data storage volume on the second block data storage system, wherein the responding to the one or more data access requests by the first program further includes initiating performance of the one or more data access requests on the created mirror copy, and wherein the automatic providing of the mirror copy after the creating of the copy on the remote archival storage systems includes maintaining the created mirror copy by performing the at least one other data access request on the mirror copy so as to modify the first block data subset stored on the second block data storage system.

23. The system of claim 20 wherein the automatic providing of the mirror copy includes creating a mirror copy of the first block data storage volume on the second block data storage system, the second block data storage system being at a distinct geographical location than the first block data storage system, the first block data subset stored on the second block data storage system including the stored block data that is further modified by the at least one other data access requests.

24. The system of claim 20 wherein the system includes a first computing system that includes at least one of the one or memories, and wherein the block data storage system manager module includes software instructions for execution by the first computing system using the at least one memory.

25. The system of claim 20 wherein the block data storage system manager module consists of one or more means for providing a block data storage service that uses multiple block data storage systems to store block data storage volumes that are created by users of the block data storage service and accessed over one or more networks on behalf of one or more executing programs associated with the users, the providing of the block data storage service including:

creating one or more block data storage volumes for use by one or more executing programs, each of the block data storage volumes being stored on one of the multiple block data storage systems;

after access to a first of the created block data storage volumes that is stored on a first of the block data storage systems is provided to a first of the one or more executing programs, responding to one or more data access requests by the first program for the first block data storage volume by initiating performance of the one or more data access requests on the first block data storage volume stored on the first block data storage system, at least one of the one or more data access requests modifying block data stored on the first block data storage volume on the first block data storage system;

after the performance of the one or more data access requests on the first block data storage volume, creating on one or more other remote archival storage systems a copy of the block data stored on the first block data storage volume; and

after the creating of the copy on the remote archival storage systems,

responding to one or more other data access requests initiated by the first program for the first block data storage volume by initiating performance of the one or more other data access requests on the first block data storage volume stored on the first block data storage system, at least one of the one or more other data access requests further modifying the block data stored on the first block data storage volume on the first block data storage system; and

automatically providing a mirror copy of the first block data storage volume on a second of the block data storage systems, the providing of the mirror copy including storing on the second block data storage system a first subset of the stored block data in the first block data storage volume and including storing on the one or more other remote archival storage systems a second subset of the stored block data in the first block data storage volume.