Consistent snapshot points in a distributed storage service

• US 10,423,609 B1
• Filed: 03/29/2017
• Issued: 09/24/2019
• Est. Priority Date: 03/29/2017
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Strauss, Jacob A.
• Primary Examiner(s): Peach, Polina G
• Application Number:
  US15/473,377
• Time in Patent Office: 909 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

Methods and apparatus for identifying consistent snapshot points for file systems in a distributed storage service in which a client may access a file system in backend storage via two or more access nodes. A snapshot coordinator initiates a two-phase commit to perform a snapshot epoch transition. The coordinator sends prepare messages to the nodes. In response, a node sends a response to the coordinator, stops issuing writes to the file system, sets a timer, and waits for a commit message. Upon receiving a response from all nodes, the coordinator sends a commit message. Upon receiving the commit or the timer expiring, the nodes increment an epoch counter and begin issuing writes to the file system tagged with the new epoch and indicating that a copy-on-write operation be performed. The transition is successful if all nodes respond to the request message; otherwise, the two-phase commit is retried.

Claims

1. A distributed storage service, comprising:

a plurality of computing devices that implement;

a storage subsystem comprising a plurality of storage nodes configured to store data for a plurality of data systems;

an access subsystem comprising a plurality of access nodes, wherein two or more of the access nodes are assigned to a particular data system of the plurality of data systems and configured to;

receive data requests for the data system; and

issue write operations to the storage subsystem to update data of the data system as indicated by the data requests, wherein each write operation indicates a current snapshot epoch of the data system; and

a snapshot coordinator implemented by one or more of the plurality of computing devices and configured to;

obtain a snapshot request for the data system to generate a new snapshot; and

perform a two-phase commit with the two or more access nodes assigned to the data system to establish a consistent snapshot point for the data system;

wherein, to perform the two-phase commit, the snapshot coordinator is configured to;

send a prepare message to each of the two or more access nodes, wherein the prepare message directs the access nodes to prepare to perform a snapshot epoch transition for the data system, wherein the snapshot epoch transition comprises incrementing the snapshot epoch at each of the two or more access nodes to thus establish a consistent snapshot point for the data system; and

upon receiving a response message to the prepare message from each of the two or more access nodes, send a commit message to each of the two or more access nodes, wherein the commit message directs the access nodes to commit to the snapshot epoch transition so that subsequent write operations issued by the access nodes to the storage subsystem indicate a new snapshot epoch.

2. The distributed storage service as recited in claim 1, wherein each of the two or more access nodes is configured to, in response to receiving the prepare message from the snapshot coordinator:

stop issuing write operations for the data system to the storage subsystem;

wait for one or more pending write operations for the data system to the storage subsystem to complete;

send a response message to the prepare message to the snapshot coordinator;

start a presumed commit timer and wait for a commit message from the snapshot coordinator; and

in response to receiving a commit message from the snapshot coordinator or to presumed commit timer expiring;

increment the snapshot epoch; and

resume issuing write operations for the data system to the storage subsystem, wherein each subsequent write operation indicates the new snapshot epoch of the data system.

3. The distributed storage service as recited in claim 2, wherein each subsequent write operation further indicates to the storage subsystem to perform copy-on-write operations for the data indicated by the write operation, wherein a copy-on-write operation stores previous snapshot epoch data to a storage location to preserve the previous snapshot epoch data when writing the new snapshot epoch data to the data system.

4. The distributed storage service as recited in claim 1, wherein the snapshot coordinator is further configured to, after receiving a response message to the prepare message from each of the two or more access nodes, initiate generation of the snapshot of the data system for a previous snapshot epoch, wherein the snapshot includes results of write operations to the data system as of the previous snapshot epoch and does not include results of the write operations of the new snapshot epoch.

5. The distributed storage service as recited in claim 4, wherein the snapshot coordinator is further configured to restart the two-phase commit with the two or more access nodes assigned to the data system in response to determining that the generation of the snapshot of the data system for the previous snapshot epoch has failed.

6. The distributed storage service as recited in claim 1, wherein the data system is a file system, wherein the data of the file system includes file system data and file system metadata, and wherein the write operations include write operations to update file system data and write operations to update file system metadata.

7. The distributed storage service as recited in claim 1, wherein the snapshot coordinator is further configured to restart the two-phase commit with the two or more access nodes assigned to the data system in response to determining that the two-phase commit has failed.

8. A method, comprising:

performing, by a snapshot coordinator for a distributed storage service implemented by one or more computer systems;

providing a plurality of access nodes of the distributed storage service, wherein the access nodes are assigned to a data system stored by a storage subsystem of the distributed storage service, the access nodes configured to perform;

receiving data requests for the data system; and

issuing write operations to the storage subsystem to update data of the data system as indicated by the data requests, wherein each write operation indicates a current snapshot epoch of the data system;

in response to a snapshot request to generate a snapshot of the data system, performing a two-phase commit with a plurality of access nodes of the distributed storage service to establish a consistent snapshot point for the data system, wherein performing the two-phase commit comprises;

sending a prepare message to each of the plurality of access nodes, wherein the prepare message directs the access nodes to prepare to perform a snapshot epoch transition for the data system, wherein performing a snapshot epoch transition comprises incrementing a snapshot epoch at each of the plurality of access nodes to thus establish a consistent snapshot point for the data system;

when a response message to the prepare message is received from each of the access nodes, sending a commit message to each of the plurality of access nodes, wherein the commit message directs the access nodes to commit to the snapshot epoch transition so that subsequent write operations issued by the access nodes to the storage subsystem indicate a new snapshot epoch; and

when a response message to the prepare message is not received from each of the access nodes, sending a prepare message to each of the plurality of access nodes to retry the two-phase commit.

9. The method as recited in claim 8, wherein the snapshot coordinator waits for a designated time period to receive the response messages from the access nodes and, if a response message to the prepare message is not received from each of the access nodes within the designated time period, sends the prepare message to each of the plurality of access nodes to retry the two-phase commit.

10. The method as recited in claim 8, wherein the plurality of access nodes are configured to perform:

stopping the issuing of write operations for the data system to the storage subsystem after receiving the prepare message;

continuing to issue read request for the data system to the storage subsystem after receiving the prepare message; and

resuming the issuing of write operations after receiving the commit message.

11. The method as recited in claim 10, further comprising the storage system updating the data of the data system as indicated by the write operations, wherein each element of data that is updated is tagged with the current snapshot epoch.

12. The method as recited in claim 10, further comprising performing, by at least one of the plurality of access nodes in response to receiving the prepare message from the snapshot coordinator:

stopping the issuing of write operations for the data system to the storage subsystem;

upon completion of pending write operations for the data system to the storage system, sending a response message to the prepare message to the snapshot coordinator;

starting a presumed commit timer and waiting for a commit message from the snapshot coordinator; and

in response to receiving a commit message from the snapshot coordinator or to presumed commit timer expiring;

incrementing the snapshot epoch; and

resuming the issuing of write operations for the data system to the storage subsystem, wherein each subsequent write operation indicates the new snapshot epoch of the data system.

13. The method as recited in claim 12, wherein each subsequent write operation further indicates to the storage subsystem to perform copy-on-write operations for the data indicated by the write operation, wherein a copy-on-write operation stores previous snapshot epoch data to a storage location to preserve the previous snapshot epoch data when writing the new snapshot epoch data to the data system.

14. The method as recited in claim 10, wherein the data system is a file system, wherein the data of the file system includes file system data and file system metadata, and wherein the write operations include write operations to update file system data and write operations to update file system metadata.

15. The method as recited in claim 14, wherein the distributed storage service further comprises a metadata subsystem comprising a plurality of metadata servers, wherein the method further comprises:

the plurality of access nodes sending the write operations to update file system metadata to the metadata servers; and

the metadata services issuing the write operations to update file system metadata to the storage subsystem.

16. The method as recited in claim 8, further comprising the snapshot coordinator, after receiving a response message to the prepare message from each of the access nodes, initiating generation of the snapshot of the data system for a previous snapshot epoch, wherein the snapshot includes results of write operations to the data system as of the previous snapshot epoch and does not include results of the write operations of the new snapshot epoch.

17. The method as recited in claim 16, further comprising the snapshot coordinator restarting the two-phase commit with the plurality of access nodes of the distributed storage service to establish a consistent snapshot point for the data system in response to determining that the generation of the snapshot of the data system for the previous snapshot epoch has failed.

18. A non-transitory computer-readable storage medium storing program instructions that when executed on one or more computers cause the one or more computers to implement:

a plurality of access nodes in a distributed storage service configured to;

receive data requests for a data system stored by a storage subsystem of the distributed storage service; and

issue write operations to the storage subsystem to update data of the data system as indicated by the data requests, wherein each write operation indicates a current snapshot epoch of the data system; and

a snapshot coordinator for the distributed storage service configured to, in response to a snapshot request to generate a snapshot of the data system, perform a two-phase commit with the access nodes to establish a consistent snapshot point for the data system, wherein, to perform the two-phase commit, the snapshot coordinator is configured to;

send a prepare message to each of the access nodes, wherein the prepare message directs the access nodes to prepare to perform a snapshot epoch transition for the data system, wherein the snapshot epoch transition comprises incrementing the snapshot epoch at each of the access nodes to thus establish a consistent snapshot point for the data system; and

upon receiving a response message to the prepare message from each of the access nodes, send a commit message to each of the access nodes, wherein the commit message directs the access nodes to commit to the snapshot epoch transition so that subsequent write operations issued by the access nodes to the storage subsystem indicate a new snapshot epoch.

19. The non-transitory computer-readable storage medium as recited in claim 18, wherein each of the access nodes is configured to, in response to receiving the prepare message from the snapshot coordinator:

stop issuing write operations for the data system to the storage subsystem;

wait for one or more pending write operations for the data system to the storage subsystem to complete;

send a response message to the prepare message to the snapshot coordinator;

start a presumed commit timer and wait for a commit message from the snapshot coordinator; and

in response to receiving a commit message from the snapshot coordinator or to presumed commit timer expiring;

increment the snapshot epoch; and

resume issuing write operations for the data system to the storage subsystem, wherein each subsequent write operation indicates the new snapshot epoch of the data system;

wherein each subsequent write operation further indicates to the storage subsystem to perform copy-on-write operations for the data indicated by the write operation, wherein a copy-on-write operation stores previous snapshot epoch data to a storage location to preserve the previous snapshot epoch data when writing the new snapshot epoch data to the data system.

20. The non-transitory computer-readable storage medium as recited in claim 18, wherein the snapshot coordinator is further configured to, after receiving a response message to the prepare message from each of the access nodes, initiate generation of the snapshot of the data system for a previous snapshot epoch, wherein the snapshot includes results of all write operations to the data system as of the previous snapshot epoch and does not include results of the write operations of the new snapshot epoch.