Method and system for automatic replication data verification and recovery

• US 10,459,632 B1
• Filed: 09/16/2016
• Issued: 10/29/2019
• Est. Priority Date: 09/16/2016
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Chen, Xiangping, Meiri, David, Stronge, William R., Shvaiger, Felix
• Primary Examiner(s): Bonzo, Bryce P
• Assistant Examiner(s): Arcos, Jeison C
• Application Number:
  US15/267,949
• Time in Patent Office: 1,138 Days
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 3/0607 by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
• CPC Class Codes: G06F 3/0616 in relation to life time, e.g. increasing Mean Time Between Failures [MTBF]
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Example embodiments of the present invention relate to methods, systems, and a computer program product for performing asynchronous replication of a snapshot. The method includes assigning a unique identifier to a snapshot at a source node in a data storage system. Asynchronous replication of the snapshot, including the unique identifier, then may be performed from the source node to a target node. Integrity of the replicated snapshot having the unique identifier then may be verified at the target node.

Claims

1. A method comprising:

assigning a unique identifier to a snapshot at a source node in a data storage system;

performing asynchronous replication of the snapshot, including the unique identifier, from the source node to a target node;

performing integrity verification on the replicated snapshot having the unique identifier at the target node to determine whether or not the replicated snapshot matches the snapshot having the unique identifier at the source node;

setting a first consistency attribute associated with the replicated snapshot to a first value responsive to the integrity verification indicating that the replicated snapshot does not match the snapshot having the unique identifier at the source node;

triggering a replication recovery in response to the first consistency attribute associated with the replication snapshot having the first value;

selecting a replication recovery snapshot from a list of candidate snapshots, a second consistency attribute associated with the selected replication recovery snapshot having a second value different from the first value, wherein the first value indicates an inconsistent verification status and the second value indicates one of a verified verification status and a synchronized but not yet verified verification status; and

performing replication recovery according to the selected replication recovery snapshot.

2. The method of claim 1

wherein the data storage system comprises a plurality of nodes;

and

wherein the unique identifier is a globally unique identifier unique across all nodes of the data storage system.

3. The method of claim 2 wherein performing integrity verification on the replicated snapshot having the unique identifier at the target node comprises:

calculating a first signature indicative of the data of the snapshot at the source node in the data storage system;

calculating a second signature indicative of the data of the snapshot at the target node in the data storage system; and

comparing the first signature calculated at the source node and the second signature calculated at the target node to identify a difference, if any, between the first and second signatures.

4. The method of claim 1 further comprising performing the integrity verification on the replication recovery snapshot by:

calculating a first signature indicative of the data of a source snapshot corresponding to the replication recovery snapshot in the data storage system;

calculating a second signature indicative of the data of the replication recovery snapshot in the data storage system;

comparing the first signature and the second signature to identify a difference, if any, between the first and second signatures; and

responsive to the first signature and the second signature matching, setting the second consistency attribute associated with the selected replication recovery snapshot to the second value to indicate that the selected replication recovery snapshot has been verified.

5. The method of claim 3 wherein one or more of calculating a first signature indicative of the data of the snapshot at the source node in the data storage system and calculating a second signature indicative of the data of the snapshot at the target node in the data storage system comprises calculating the signature according to a value selected from a group consisting of:

disk data;

a plurality of signatures of portions of the disk data; and

a plurality of short hash handles.

6. The method of claim 5 wherein the plurality of signatures of portions of the disk data and the plurality of short hash handles are stored in memory in the data storage system.

7. The method of claim 3 wherein the triggering of the replication recovery is performed responsive to the first signature calculated at the source node and the second signature calculated at the target node not matching.

8. The method of claim 1 wherein performing replication recovery according to a selected replication recovery snapshot comprises selecting the replication recovery snapshot from the list of candidate snapshots according to the value of the second consistency attribute and a replication recovery policy associated with the data storage system.

9. The method of claim 3 wherein performing the integrity verification on the replicated snapshot having the unique identifier at the target node comprises performing the integrity verification on the replicated snapshot as a background process of the data storage system.

10. A data storage system comprising:

a source node;

a target node;

one or more processors; and

memory storing computer program code that when executed on the one or more processors performs the operations of;

assigning a unique identifier to a snapshot at a source node in a data storage system;

performing asynchronous replication of the snapshot, including the unique identifier, from the source node to a target node;

performing integrity verification on the replicated snapshot having the unique identifier at the target node to determine whether or not the replicated snapshot matches the snapshot having the unique identifier at the source node;

setting a first consistency attribute associated with the replicated snapshot to a first value responsive to the integrity verification indicating that the replicated snapshot does not match the snapshot having the unique identifier at the source node;

triggering a replication recovery in response to the first consistency attribute associated with the replication snapshot having the first value;

selecting a replication recovery snapshot from a list of candidate snapshots, a second consistency attribute associated with the selected replication recovery snapshot having a second value different from the first value, wherein the first value indicates an inconsistent verification status and the second value indicates one of a verified verification status and a synchronized but not yet verified verification status; and

performing replication recovery according to the selected replication recovery snapshot.

11. The system of claim 10

wherein the data storage system comprises a plurality of nodes;

and

wherein the unique identifier is a globally unique identifier unique across all nodes of the data storage system.

12. The system of claim 11 wherein performing integrity verification on the replicated snapshot having the unique identifier at the target node comprises:

calculating a first signature indicative of the data of the snapshot at the source node in the data storage system;

calculating a second signature indicative of the data of the snapshot at the target node in the data storage system; and

comparing the first signature calculated at the source node and the second signature calculated at the target node to identify a difference, if any, between the first and second signatures.

13. The system of claim 12 further comprising performing integrity verification on the replication recovery snapshot by:

calculating a first signature indicative of the data of a source snapshot corresponding to the replication recovery snapshot in the data storage system;

calculating a second signature indicative of the data of the replication recovery snapshot in the data storage system;

comparing the first signature and the second signature to identify a difference, if any, between the first and second signatures; and

responsive to the first signature and the second signature matching, setting the second consistency attribute associated with the selected replication recovery snapshot to the second value to indicate that the selected replication recovery snapshot has been verified.

14. The system of claim 12 wherein one or more of calculating a first signature indicative of the data of the snapshot at the source node in the data storage system and calculating a second signature indicative of the data of the snapshot at the target node in the data storage system comprises calculating the signature according to a value selected from a group consisting of:

disk data;

a plurality of signatures of portions of the disk data; and

a plurality of short hash handles.

15. The system of claim 14 wherein the plurality of signatures of portions of the disk data and the plurality of short hash handles are stored in memory in the data storage system.

16. The system of claim 12 wherein the triggering of the replication recovery is performed responsive to the first signature calculated at the source node and the second signature calculated at the target node not matching.

17. The system of claim 10 wherein performing replication recovery according to a selected replication recovery snapshot comprises selecting the replication recovery snapshot from the list of candidate snapshots according to the value of the second consistency attribute and a replication recovery policy associated with the data storage system.

18. The system of claim 12 wherein performing the integrity verification on the replicated snapshot having the unique identifier at the target node comprises performing the integrity verification on the replicated snapshot as a background process of the data storage system.

19. A computer program product including a non-transitory computer readable storage medium having computer program code encoded there on that when executed on a processor of a computer causes the computer to verify integrity of snapshots, comprising:

computer program code for assigning a unique identifier to a snapshot at a source node in a data storage system;

computer program code for performing asynchronous replication of the snapshot, including the unique identifier, from the source node to a target node;

computer program code for performing integrity verification on the replicated snapshot having the unique identifier at the target node to determine whether or not the replicated snapshot matches the snapshot having the unique identifier at the source node;

computer program code for setting a first consistency attribute associated with the replicated snapshot to a first value responsive to the integrity verification indicating that the replicated snapshot does not match the snapshot having the unique identifier at the source node;

computer program code for triggering a replication recovery in response to the first consistency attribute associated with the replication snapshot having the first value;

computer program code for selecting a replication recovery snapshot from a list of candidate snapshots, a second consistency attribute associated with the selected replication recovery snapshot having a second value different from the first value, wherein the first value indicates an inconsistent verification status and the second value indicates one of a verified verification status and a synchronized but not yet verified verification status; and

computer program code for performing replication recovery according to the selected replication recovery snapshot.

20. The computer program product of claim 19 further comprising performing the integrity verification on the replication recovery snapshot by:

calculating a first signature indicative of the data of a source snapshot corresponding to the replication recovery snapshot in the data storage system;

calculating a second signature indicative of the data of the replication recovery snapshot in the data storage system;

comparing the first signature and the second signature to identify a difference, if any, between the first and second signatures; and

responsive to the first signature and the second signature matching, setting the second consistency attribute associated with the selected replication recovery snapshot to the second value to indicate that the selected replication recovery snapshot has been verified.

21. The computer program product of claim 19 wherein performing replication recovery according to a selected replication recovery snapshot comprises selecting the replication recovery snapshot from the list of candidate snapshots according to the value of the second consistency attribute and a replication recovery policy associated with the data storage system.