SAND TIMER ALGORITHM FOR TRACKING IN-FLIGHT DATA STORAGE REQUESTS FOR DATA REPLICATION

• US 20200125649A1
• Filed: 10/17/2018
• Published: 04/23/2020
• Est. Priority Date: 10/17/2018
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Hu, Ying, Chen, Xiangping
• Granted Patent: US 10,642,788 B1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/1815 Journaling file systems
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A method is disclosed comprising: generating a first snapshot of a first storage subsystem; detecting, by a management node, that all in-flight data storage requests recorded in drain tables of storage nodes in the first storage subsystem have been completed, the in-flight data storage requests recorded in the drain tables of the storage nodes being replicated in a second storage subsystem; causing, by the management node, each of the storage nodes to flip the respective designations of the tracking tables in the node'"'"'s respective pair of tracking tables; and transmitting, from the management node to the second storage subsystem, an instruction which when received by the second storage subsystem causes the second storage subsystem to generate a second snapshot of the second storage subsystem.

Claims

1. A method for use in a first storage subsystem having a management node and a plurality of storage nodes, each of the storage nodes having a respective pair of tracking tables for recording in-flight data storage requests, the tracking tables in each pair having different respective designations, the tracking tables in each pair alternating between a drain table designation and an active table designation, the method comprising:

generating a first snapshot of the first storage subsystem;

detecting, by the management node, that all in-flight data storage requests recorded in the drain tables of the storage nodes have been completed, the in-flight data storage requests recorded in the drain tables of the storage nodes being replicated in a second storage subsystem;

causing, by the management node, each of the plurality of storage nodes to flip the respective designations of the tracking tables in the node'"'"'s respective pair of tracking tables; and

transmitting, from the management node to the second storage subsystem, an instruction which when received by the second storage subsystem causes the second storage subsystem to generate a second snapshot of the second storage subsystem, the instruction being transmitted after the first snapshot is generated and all in-flight data storage requests recorded in the drain tables of the storage nodes have been completed.

2. The method of claim 1, further comprising:

detecting that a replication session of the first storage subsystem and the second storage subsystem is out of sync; and

transmitting to the second storage subsystem a difference between at least one production object of the first storage subsystem and the first snapshot of the first storage subsystem.

3. The method of claim 1, wherein the in-flight data storage requests recorded in the drain tables of the storage nodes are synchronously replicated to the second storage subsystem.

4. The method of claim 1, wherein detecting that all in-flight data storage requests recorded in the drain tables of the storage nodes have been completed includes receiving a different respective message from each one the storage nodes indicating that all data storage requests recorded in the storage node'"'"'s respective drain table have been completed.

5. The method of claim 1, wherein causing each of the storage nodes to flip the respective designations of the tracking tables in the node'"'"'s respective pair of tracking tables includes transmitting to each of the plurality of storage nodes a respective instruction which when received by the storage node causes the storage node to:

(i) designate the storage node'"'"'s current active table as drain table, and (ii) designate the storage node'"'"'s current drain table as active table.

6. The method of claim 1, wherein recording an in-flight data storage request in any of the tracking tables includes adding an identifier corresponding to the in-flight data storage request to the tracking table, such that identifiers corresponding to in-flight data storage requests are added to any given one of the tracking tables only during periods when the given tracking table carries active table designation.

7. (canceled)

8. An apparatus for use in a first storage subsystem having a plurality of storage nodes, each of the storage nodes having a respective pair of tracking tables for recording in-flight data storage requests, the tracking tables in each pair having different respective designations, the tracking tables in each pair alternating between a drain table designation and an active table designation, the apparatus comprising:

a memory; and

at least one processor operatively coupled to the memory, the at least one processor being configured to;

generate a first snapshot of the first storage subsystem;

detect that all in-flight data storage requests recorded in the drain tables of the storage nodes have been completed, the in-flight data storage requests recorded in the drain tables of the storage nodes being replicated in a second storage subsystem;

cause each of the plurality of storage nodes to flip the respective designations of the tracking tables in the node'"'"'s respective pair of tracking tables; and

transmit, to the second storage subsystem, an instruction which when received by the second storage subsystem causes the second storage subsystem to generate a second snapshot of the second storage subsystem, the instruction being transmitted after the first snapshot is generated and all in-flight data storage requests recorded in the drain tables of the storage nodes have been completed.

9. The apparatus of claim 8, wherein the at least one processor is further configured to:

detect that a replication session of the first storage subsystem and the second storage subsystem is out of sync; and

transmit to the second storage subsystem a difference between at least one production object of the first storage subsystem and the first snapshot of the first storage subsystem.

10. The apparatus of claim 8, wherein the in-flight data storage requests recorded in the drain tables of the storage nodes are synchronously replicated to the second storage subsystem.

11. The apparatus of claim 8, wherein detecting that all in-flight data storage requests recorded in the drain tables of the storage nodes have been completed includes receiving a different respective message from each one the storage nodes indicating that all data storage requests recorded in the storage node'"'"'s respective drain table have been completed.

12. The apparatus of claim 8, wherein causing each of the storage nodes to flip the respective designations of the tracking tables in the node'"'"'s respective pair of tracking tables includes transmitting to each of the plurality of storage nodes a respective instruction which when received by the storage node causes the storage node to:

(i) designate the storage node'"'"'s current active table as drain table, and (ii) designate the storage node'"'"'s current drain table as active table.

13. The apparatus of claim 8, wherein recording an in-flight data storage request in any of the tracking tables includes adding an identifier corresponding to the in-flight data storage request to the tracking table, such that identifiers corresponding to in-flight data storage requests are added to any given one of the tracking tables only during periods when the given tracking carries the active table designation.

14. (canceled)

15. A non-transitory computer-readable medium configured to store one or more processor-executable instructions which when executed by at least one processor cause the at least one processor to perform the operations of:

identifying a plurality of storage nodes in a first storage subsystem, each of the storage nodes having a respective pair of tracking tables for recording in-flight data storage requests, the tracking tables in each pair having different respective designations, the tracking tables in each pair alternating between a drain table designation and an active table designation;

generating a first snapshot of the first storage subsystem;

detecting that all in-flight data storage requests recorded in drain tables of the storage nodes have been completed, the in-flight data storage requests recorded in the drain tables of the storage nodes being replicated in a second storage subsystem;

causing each of the plurality of storage nodes to flip the respective designations of the tracking tables in the node'"'"'s respective pair of tracking tables; and

transmitting to the second storage subsystem, an instruction which when received by the second storage subsystem causes the second storage subsystem to generate a second snapshot of the second storage subsystem, the instruction being transmitted after the first snapshot is generated and all in-flight data storage requests recorded in the drain tables of the storage nodes have been completed.

16. The non-transitory computer-readable medium of claim 15, wherein the processor-executable instructions, when executed by the at least one processor, further cause the at least one processor to perform the operations of:

detecting that a replication session of the first storage subsystem and the second storage subsystem is out of sync; and

transmitting to the second storage subsystem a difference between at least one production object of the first storage subsystem and the first snapshot of the first storage subsystem.

17. The non-transitory computer-readable medium of claim 15, wherein the in-flight data storage requests recorded in the drain tables of the storage nodes are synchronously replicated to the second storage subsystem.

18. The non-transitory computer-readable medium of claim 15, wherein detecting that all in-flight data storage requests recorded in the drain tables of the storage nodes have been completed includes receiving a different respective message from each one the storage nodes indicating that all data storage requests recorded in the storage node'"'"'s respective drain table have been completed.

19. The non-transitory computer-readable medium of claim 15, wherein causing each of the storage nodes to flip the respective designations of the tracking tables in the node'"'"'s respective pair of tracking tables includes transmitting to each of the plurality of storage nodes a respective instruction which when received by the storage node causes the storage node to:

(i) designate the storage node'"'"'s current active table as drain table, and (ii) designate the storage node'"'"'s current drain table as active table.

20. The non-transitory computer-readable medium of claim 15, wherein recording an in-flight data storage request in any of the tracking tables includes adding an identifier corresponding to the in-flight data storage request to the tracking table, such that identifiers corresponding to in-flight data storage requests are added to any given one of the tracking tables only during periods when the given tracking table carries the active table designation.