Fast failover with multiple secondary nodes

• US 8,689,043 B1
• Filed: 06/30/2003
• Issued: 04/01/2014
• Est. Priority Date: 06/30/2003
• Status: Expired due to Fees

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Operating Corporation (NortonLifeLock Inc.)
• Inventors: Bezbaruah, Angshuman, Kekre, Anand A., Pendharkar, Niranjan S.
• Primary Examiner(s): Blair, April Y
• Assistant Examiner(s): Cygiel, Gary W
• Application Number:
  US10/610,139
• Time in Patent Office: 3,928 Days
• Field of Search: 711/148, 711162-165, 714/4, 714/4.11, 714/6.3, 714/6.32
• US Class Current: 714/6.3
• CPC Class Codes: G06F 11/2041 with more than one idle spare processing component
• CPC Class Codes: G06F 11/2058 using more than 2 mirrored copies
• CPC Class Codes: G06F 11/2064 while ensuring consistency
• CPC Class Codes: G06F 11/2074 Asynchronous techniques
• CPC Class Codes: G06F 11/2076 Synchronous techniques
• CPC Class Codes: G06F 11/2082 Data synchronisation

Abstract

A method, system, computer system, and computer program product to quickly synchronize replicas of data stored at multiple secondary nodes upon failure of a primary node. Secondary nodes to which an update to data in a region of a storage area is sent are identified. When all of the secondary nodes have acknowledged the update, a notification is sent to each of the secondary nodes. Respective indicator(s) (e.g., “outstanding update” or “out of sync” indicators) can be set to indicate that the data have changed in respective region(s) at one of the secondary nodes when the update has been received at that secondary node. The indicator(s) can be reset for the respective region(s) when the notification sent by the primary node is received at the secondary node. Data in the region(s) can be synchronized with data at secondary nodes when either node has an indicator set for that region.

Claims

1. A method comprising:

identifying a plurality of secondary nodes;

sending an update to each of the plurality of secondary nodes, wherein

the update is sent from a primary node, and

the update is an update to data in a region of a storage area;

setting an indicator for the each of the plurality of the secondary nodes, wherein

the setting causes the indicator to indicate that the update has been sent to the each of the plurality of secondary nodes;

determining that a failure has occurred at the primary node; and

in response to the determining, selecting a secondary node of the plurality of secondary nodes as a new primary node, wherein

the selecting is based on the indicators for the each of the plurality of secondary nodes.

2. The method of claim 1, wherein

the plurality of secondary nodes comprises all secondary nodes.

3. The method of claim 1, further comprising:

in response to the sending, incrementing a regional counter stored on the primary node by a number of secondary nodes to which the update is sent, wherein

the regional counter is a number of secondary nodes from which an acknowledgement to the update has not been received;

in response to receiving an acknowledgement from a secondary node among the plurality of secondary nodes to which the update is sent, decrementing the regional counter;

in response to the regional counter reaching a value prior to the incrementing, determining that all of the plurality of secondary nodes have acknowledged the update; and

sending a notification to the each of the plurality of secondary nodes once all of the plurality of secondary nodes have acknowledged the update.

4. The method of claim 3, further comprising:

setting a respective indicator in a respective region of a respective storage area at one of the plurality of secondary nodes when the update is received by the one secondary node; and

resetting the respective indicator when the notification is received at the one secondary node.

5. The method of claim 4, further comprising:

synchronizing first data in the respective region with second data in a second respective region of a second respective storage area at a second secondary node of the plurality of secondary nodes when at least one of the respective indicator and a second respective indicator for the second respective region is set.

6. The method of claim 5, wherein

the synchronizing does not affect third data in a third region of the second respective storage area when the respective indicator is not set and a third respective indicator for the third respective region is not set.

7. The method of claim 4, further comprising:

synchronizing first data in the respective region with second data in a second respective region of a second respective storage area at the secondary node selected as the new primary node wherein

at least one of the respective indicator and a second respective indicator for the second respective region is set.

8. The method of claim 3, wherein the setting further comprises:

setting a sent indicator for the region for the each of the plurality of secondary nodes when the update is sent to the each of the plurality of secondary nodes.

9. The method of claim 8, further comprising:

setting a received indicator for the region for the each of the plurality of secondary nodes when an acknowledgement of the update is received from the each of the plurality of secondary nodes.

10. The method of claim 9, wherein

the sending the notification to the each of the plurality of secondary nodes comprises determining that a respective sent indicator and a respective received indicator for the region are set for the each of the plurality of secondary nodes.

11. The method of claim 3, further comprising:

clearing a respective indicator for the region on one of the plurality of secondary nodes when the one secondary node receives the notification.

12. The method of claim 1, wherein

the secondary node selected as the new primary node is the most recently updated secondary node of the plurality of secondary nodes.

13. The method of claim 1, further comprising:

identifying a second update, wherein

the second update is an update to second data in a second region of a second storage area at the new primary node; and

in response to the identifying the second update, synchronizing the second data in the second region with third data in a third respective region of a third storage area at each of the remaining secondary nodes of the plurality of secondary nodes.

14. A non-transitory computer-readable storage medium comprising computer-readable instructions, wherein the computer-readable instructions, when executed by a processor, are configured to perform a method comprising:

identifying a plurality of secondary nodes;

sending an update to each of the plurality of secondary nodes, wherein

the update is sent by a primary node, and

the update is an update to data in a region of a storage area;

setting an indicator for the each of the plurality of secondary nodes, wherein

the setting causes the indicator to indicate that the update has been sent to the each of the plurality of secondary nodes;

determining that a failure has occurred at the primary node; and

in response to the determining, selecting a secondary node of the plurality of secondary nodes as a new primary node, wherein

the selecting is based on the indicators for the each of the plurality of secondary nodes.

15. The non-transitory computer-readable storage medium of claim 14, wherein the method further comprises:

in response to the sending, incrementing a regional counter stored on the primary node by a number of secondary nodes to which the update is sent, wherein

the regional counter is a number of secondary nodes from which an acknowledgement to the update has not been received;

in response to receiving an acknowledgement from a secondary node among the plurality of secondary nodes to which the update is sent, decrementing the regional counter;

in response to the regional counter reaching a value prior to the incrementing, determining that all of the plurality of secondary nodes have acknowledged the update; and

sending a notification to the each of the plurality of secondary nodes once all of the plurality of secondary nodes have acknowledged the update.

16. The non-transitory computer-readable storage medium of claim 15, wherein the method further comprises:

setting a respective indicator in a respective region of a respective storage area at one of the plurality of secondary nodes when the update is received by the one secondary node; and

resetting the respective indicator when the notification is received at the one secondary node.

17. The non-transitory computer-readable storage medium of claim 14, wherein the method further comprises:

synchronizing first data in the respective region with second data in a second respective region of a second respective storage area at a second secondary node of the plurality of secondary nodes when at least one of the respective indicator and a second respective indicator for the second respective region is set.

18. The non-transitory computer-readable storage medium of claim 17, wherein

the synchronizing does not affect third data in a third region of the second respective storage area when the respective indicator is not set and a third respective indicator for the third respective region is not set.

19. The non-transitory computer-readable storage medium of claim 14, wherein

the secondary node selected as the new primary node is the most recently updated secondary node of the plurality of secondary nodes.

20. The non-transitory computer-readable storage medium of claim 14, wherein the method further comprises:

identifying a second update, wherein

the second update is an update to second data in a second region of a second storage area at the new primary node; and

in response to the identifying the second update, synchronizing the second data in the second region with third data in a third respective region of a third storage area at each of the remaining secondary nodes of the plurality of secondary nodes.

21. A computer system comprising:

one or more processors;

a non-transitory computer-readable storage medium coupled to said one or more processors; and

a plurality of instructions, wherein

said plurality of instructions are encoded in said non-transitory computer-readable storage medium, and

said plurality of instructions, when executed by said one or more processors, are configured to perform a method comprising

identifying a plurality of secondary nodes,

sending an update to each of the plurality of secondary nodes, wherein

the update is sent from a primary node, and

the update is an update to data in a region of a storage area,

setting an indicator for the each of the plurality of secondary nodes, wherein

the setting causes the indicator to indicate that the update has been sent to the each of the plurality of secondary nodes,

determining that a failure has occurred at the primary node, and

in response to the determining, selecting a secondary node of the plurality of secondary nodes as a new primary node, wherein

the selecting is based on the indicators for the each of the plurality of secondary nodes.

22. The computer system of claim 21, wherein the method further comprises:

incrementing a regional counter stored on the primary node by a number of secondary nodes to which the update is sent, in response to the sending, wherein

the regional counter is a number of secondary nodes from which an acknowledgement to the update has not been received,

decrementing the regional counter in response to receiving an acknowledgement from a secondary node among the plurality of secondary nodes to which the update is sent,

determining that all of the plurality of secondary nodes have acknowledged the update, in response to the regional counter reaching a value prior to the incrementing, and

sending a notification to each of the plurality of secondary nodes once all of the plurality of secondary nodes have acknowledged the update.

23. The computer system of claim 22, wherein the method further comprises:

setting a respective indicator in a respective region of a respective storage area at one of the plurality of secondary nodes when the update is received by the one secondary node, and

resetting the respective indicator when the notification is received at the one secondary node.

24. The computer system of claim 23, wherein the method further comprises:

synchronizing first data in the respective region with second data in a second respective region of a second respective storage area at a second secondary node of the plurality of secondary nodes when at least one of the respective indicator and a second respective indicator for the second respective region is set.

25. The computer system of claim 24, wherein

the synchronizing does not affect third data in a third region of the second respective storage area when the respective indicator is not set and a third respective indicator for the third respective region is not set.

26. The computer system of claim 21, wherein

the secondary node selected as the new primary node is the most recently updated secondary node of the plurality of secondary nodes.

27. The computer system of claim 21, wherein the method further comprises:

identifying a second update, wherein

the second update is an update to second data in a second region of a second storage area at the new primary node, and

synchronizing the second data in the second region with third data in a third respective region of a third storage area at each of the remaining secondary nodes of the plurality of secondary nodes, in response to the identifying the second update.