Reconciliation in sync replication

• US 9,715,433 B2
• Filed: 08/29/2014
• Issued: 07/25/2017
• Est. Priority Date: 08/29/2014
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Mu, Yuedong, Ngan, Ching-Yuk Paul, Sundararajan, Manoj V.
• Primary Examiner(s): PATEL, KAMINI B
• Application Number:
  US14/472,885
• Publication Number:
  US 20160062836A1
• Time in Patent Office: 1,061 Days
• Field of Search: 714 63, 714 632
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/2058 using more than 2 mirrored copies
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2076 Synchronous techniques
• CPC Class Codes: G06F 11/2089 Redundant storage control functionality
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

A distributed storage system replicates data for a primary logical storage object on a primary node of the storage system to a secondary logical storage object on a secondary node on the distributed storage system. Failures in writing data to the primary logical storage object or failures in the replication of the data to the secondary logical storage object can cause data that should be synchronized to become divergent. In cases where the data may be divergent, reconciliation operations can be performed to resynchronize the data.

Claims

1. A method comprising:

receiving, by a first node, a first request to write data to a first storage unit coupled to the first node;

performing a first write operation to the first storage unit;

replicating the first request to a second node as a replicated first request, wherein the replicated first request causes the second node to perform a second write operation to write the data to a second storage unit coupled to the second node;

performing a failure reconciliation based upon the first write operation failing, the failure reconciliation comprising;

reading old data from a location within the first storage unit targeted by the first write operation, wherein the old data was previously received by the first node from the second node; and

sending a reconciliation request, comprising the old data, to instruct the second node to overwrite the data, written to the second storage unit, using the old data;

receiving a second request to abort the first request;

performing a first abort operation on the first node based upon the second request;

forwarding the second request to the second node to invoke the second node to perform a second abort operation;

determining a combined response based upon a first result of the first write operation and a second result of the second write operation; and

performing a reconciliation operation in accordance with the combined response.

2. The method of claim 1, comprising:

determining that data at the first storage unit is different from data at the second storage unit; and

performing a reconciliation to synchronize the data at the first storage unit and the data at the second storage unit.

3. The method of claim 1, wherein old data overwrites data, stored on the second storage unit, based upon the reconciliation operation.

4. The method of claim 1, comprising:

determining that the second write operation failed to write the data to the second storage unit.

5. The method of claim 4, comprising:

terminating data replication between the first node and the second node.

6. The method of claim 1, comprising:

determining that the first abort operation failed and the second abort operation succeeded;

reading first data from the first storage unit; and

forwarding the first data to the second node to invoke the second node to overwrite second data on the second storage unit with the first data.

7. The method of claim 1, comprising:

determining that the first abort operation succeeded and the second abort operation failed;

reading first data from the first storage unit; and

forwarding the first data to the second node to invoke the second node to overwrite second data on the second storage unit with the first data.

8. A non-transitory machine readable medium having stored thereon instructions for performing a method comprising program code which when executed by a machine, causes the machine to:

receive, by a first node, a first request to write data to a first storage unit coupled to the first node;

perform a first write operation to the first storage unit;

replicate the first request to a second node as a replicated first request, wherein the replicated first request causes the second node to perform a second write operation to write the data to a second storage unit coupled to the second node;

receive, by the first node, a second request to abort the first request;

perform a first abort operation on the first node based upon the second request;

forward the second request to the second node to invoke the second node to perform a second abort operation;

determine a combined response based upon results of the first abort operation and the second abort operation; and

perform a reconciliation operation in accordance with the combined response.

9. The non-transitory machine readable medium of claim 8, wherein the program code causes the machine to:

determine that the first write operation failed to write the data; and

read, by the first node, old data from a location targeted by the first write operation and forward the old data to the second node.

10. The non-transitory machine readable medium of claim 9, wherein the program code causes the machine to:

instruct the second node to write the old data, wherein the old data overwrites data stored on the second storage unit.

11. The non-transitory machine readable medium of claim 8, wherein the program code causes the machine to:

determine that the second write operation failed to write the data to the second storage unit; and

terminate data replication between the first node and the second node.

12. The non-transitory machine readable medium of claim 8, wherein the first request is to write the data for a first logical storage object at the first storage unit and the second write operation is to write the data for a second logical storage object at the second storage unit.

13. The non-transitory machine readable medium of claim 8, wherein the program code causes the machine to:

determine that the first abort operation failed and the second abort operation succeeded;

read first data from the first storage unit; and

forward the first data to the second node to invoke the second node to overwrite second data on the second storage unit with the first data.

14. The non-transitory machine readable medium of claim 8, wherein the program code causes the machine to:

determine that the first abort operation succeeded and the second abort operation failed;

read first data from the first storage unit; and

forward the first data to the second node to invoke the second node to overwrite second data on the second storage unit with the first data.

15. A device comprising:

a memory containing machine readable medium comprising program code having stored thereon instructions for performing a method; and

a processor coupled to the memory, the processor configured to execute the program code to cause the processor to;

receive, by a first node, a first request to write data for a first logical storage object to a first storage unit coupled to the first node;

perform a first write operation to a first location on the first storage unit;

replicate the first request to a second node as a replicated first request, wherein the replicated first request causes the second node to perform a second write operation to write the data to a second location for a second logical storage object on a second storage unit coupled to the second node;

perform a failure reconciliation based upon the first write operation failing, the failure reconciliation comprising;

reading old data from a location within the first storage unit targeted by the first write operation, wherein the old data was previously received by the first node from the second node; and

sending a reconciliation request, comprising the old data, to instruct the second node to overwrite the data, written to the second storage unit, using the old data;

receive a second request to abort the first request;

perform a first abort operation on the first node based upon the second request;

forward the second request to the second node to invoke the second node to perform a second abort operation;

determine a combined response based upon a first result of the first write operation and a second result of the second write operation; and

perform a reconciliation operation in accordance with the combined response.

16. The device of claim 15, wherein the program code causes the processor to:

determine that data at the first location is different from data at the second location; and

perform a reconciliation to synchronize the data at the first location and the data at the second location.

17. The device of claim 15, wherein the program code causes the processor to:

instruct the second node to write old data, wherein the old data overwrites the data stored at the second location on the second storage unit.

18. The device of claim 15, wherein the program code causes the processor to:

determine that the second write operation failed to write the data to the second storage unit.

19. The device of claim 18, wherein the program code causes the processor to:

terminate data replication between the first node and the second node.

20. The device of claim 15, wherein the program code causes the processor to:

determine that the first abort operation failed and the second abort operation succeeded;

read first data from the first storage unit; and

forward the first data to the second node to invoke the second node to overwrite second data on the second storage unit with the first data.