Synchronization of snapshots in a distributed storage system

• US 9,912,748 B2
• Filed: 01/12/2015
• Issued: 03/06/2018
• Est. Priority Date: 01/12/2015
• Status: Active Grant

• Current Assignee: Mellanox Technologies Limited (NVIDIA Corporation)
• Original Assignee: Stratoscale Ltd.
• Inventors: Lazar, Aharon, Feldmann, Yael
• Primary Examiner(s): Avellino, Joseph E
• Assistant Examiner(s): Book, Phyllis A
• Application Number:
  US14/594,188
• Publication Number:
  US 20160205182A1
• Time in Patent Office: 1,149 Days
• Field of Search: None
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method includes storing data generated in a source node by sending write requests to multiple destination nodes. The destination nodes are requested to create snapshots of the data. The write requests are marked at the source node with marks that indicate to each destination node which of the write requests are pre-snapshot write requests that were issued before a snapshot request for a snapshot that the destination node is currently storing, and which of the write requests are post-snapshot write requests that were issued after the snapshot request for the snapshot that the destination node is currently storing. The snapshots are synchronized with one another at the destination nodes based on the marks.

Claims

1. A method, comprising:

generating data, which belongs to a logical volume, in a source node;

preparing at the source node, write requests associated with the logical volume, for transmission from the source node to respective destination nodes, wherein the write requests include write requests for storing replicas of the same data directed to different destination nodes;

at the source node, marking the write requests with marks that indicate to which snapshot the write request relates;

sending the write requests with the markings to the respective destination nodes;

sending from the source node to the destination nodes, a request to create a snapshot of the data; and

at the destination nodes;

receiving the write requests;

distinguishing between pre-snapshot write requests and post-snapshot write requests based on the marks in the write requests, irrespective of an order-of-arrival of the write requests at the destination nodes;

individually in each of the destination nodes, executing only the pre-snapshot write requests; and

in each destination node, queuing post-snapshot write requests until arrival of a subsequent snapshot request.

2. The method according to claim 1, comprising in each destination node, queuing the received write requests and executing only the queued write requests that become pre-snapshot write requests.

3. The method according to claim 1, wherein marking the write requests comprises tagging each write request with a respective counter value indicating the snapshot that was most recently requested by the source node.

4. The method according to claim 3, wherein synchronizing the snapshots comprises, in each destination node, distinguishing between the pre-snapshot write requests and the post-snapshot write requests by comparing the respective counter value of each write request with a reference counter value of the snapshot that the destination node is currently storing.

5. The method according to claim 1, wherein sending the request to create a snapshot comprises receiving, by the source node, a request for generating a snapshot, determining write requests that were sent to destination nodes that should be included in the snapshot, and sending the request to create the snapshot only after receiving acknowledgements for the write requests determined to be included in the snapshot.

6. A system, comprising at least a source node and multiple destination nodes,

wherein the source node is configured to generate data, which belongs to a logical volume, to prepare at the source node write requests associated with the logical volume, for transmission from the source node to respective destination nodes, wherein the write requests include write requests for storing replicas of the same data directed to different destination nodes, to mark the write requests with marks that indicate to which snapshot the write request relates, to send the write requests with the markings to the respective destination nodes, and to send from the source node to the destination nodes, a request to create a snapshot of the data;

and

wherein the destination nodes are configured to receive the write requests, to distinguish between the pre-snapshot write requests and the post-snapshot write requests based on the marks in the write requests, irrespective of an order-of-arrival of the write requests at the destination nodes,

to execute only the pre-snapshot write requests, and to queue post-snapshot write requests until arrival of a subsequent snapshot request.

7. The system according to claim 6, wherein each destination node is configured to queue the received write requests and to execute only the queued write requests that become pre-snapshot write requests.

8. The system according to claim 6, wherein the source node is configured to mark each write request with a respective counter value indicating the snapshot that was most recently requested by the source node.

9. The system according to claim 8, wherein each destination node is configured to distinguish between the pre-snapshot write requests and the post-snapshot write requests by comparing the respective counter value of each write request with a reference counter value of the snapshot that the destination node is currently storing.

10. The system according to claim 6, wherein the source node is configured to receive a request for generating a snapshot, to determine write requests that were sent to destination nodes that should be included in the snapshot, and to send the request to create the snapshot only after receiving acknowledgements for the write requests determined to be included in the snapshot.

11. A computer software product, the product comprising a tangible non-transitory computer-readable medium in which program instructions are stored, including instructions, when read by processors of a source node and multiple destination nodes that are interconnected by a communication network, cause a processor of the source node to generate data, which belongs to a logical volume, to send write requests associated with the logical volume, for transmission from the source node to respective destination nodes, wherein the write requests include write requests for storing replicas of the same data directed to different destination nodes, to mark the write requests at the source node with marks that indicate to which snapshot the write request relates, to send the write requests with the markings to the respective destination nodes, and to send from the source node to the destination nodes, a request to create a snapshot of the data;

and

including instructions which when executed by processors of the destination nodes causes the destination nodes to

receive the write requests, distinguish between pre-snapshot write requests and post-snapshot write requests based on the marks in the write requests, irrespective of an order-of-arrival of the write requests at the destination nodes,

to execute only the pre-snapshot write requests, and to queue post-snapshot write requests until arrival of a subsequent snapshot request.

12. The computer software product according to claim 11, wherein the instructions cause the processor of the source node to receive a request for generating a snapshot, to determine write requests that were sent to destination nodes that should be included in the snapshot, and to send the request to create the snapshot only after receiving acknowledgements for the write requests determined to be included in the snapshot.