Fast failover with multiple secondary nodes

• US 7,617,369 B1
• Filed: 12/04/2003
• Issued: 11/10/2009
• Est. Priority Date: 06/30/2003
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Symantec Operating Corporation (Gen Digital Inc.)
• Inventors: Bezbaruah, Angshuman, Pendharkar, Niranjan S., Kekre, Anand A.
• Primary Examiner(s): Chace; Christian P
• Assistant Examiner(s): Cygiel; Gary W
• Application Number:
  US10/728,027
• Time in Patent Office: 2,168 Days
• Field of Search: None
• US Class Current: 711/162
• CPC Class Codes: G06F 11/2074 Asynchronous techniques
• CPC Class Codes: G06F 11/2076 Synchronous techniques
• CPC Class Codes: G06F 11/2082 Data synchronisation

Abstract

A method, system, computer system, and computer program product to quickly synchronize replicas of data stored at multiple secondary nodes upon failure of a primary node. Secondary nodes to which an update to data is sent are identified. The secondary nodes insert the update into a respective log of updates to a respective copy of the data. When all of the secondary nodes have acknowledged the update, a notification is sent to each of the secondary nodes. In response to receiving the notification, each of the secondary nodes clears the update from the respective log. Data at one secondary node can be synchronized with data at another secondary node when one of the nodes has received updates from the primary node that the other secondary node has not yet received.

Claims

1. A method comprising:

identifying a plurality of secondary nodes;

sending an update from a primary node to the plurality of secondary nodes, wherein

the update identifies at least one secondary node of the plurality of secondary nodes to which the update will be sent;

in response to receiving the update from the primary node, causing the at least one secondary node to insert the update in a respective log of updates, wherein

each of the respective log of updates corresponds to a respective copy of the data, and

the inserting the update in the respective log of updates is performed at the secondary node;

in response to inserting the update in the respective log of updates, causing the update to be copied from the respective log of updates to a storage area at the least one secondary node;

sending an acknowledgement from the at least one secondary node to the primary node, wherein

the acknowledgement indicates that the update has been received at the least one secondary node;

determining that all of the plurality of secondary nodes have acknowledged the update; and

in response to the determining, causing each secondary node of the at least one secondary node to clear the update from the respective log of updates by sending a notification to each of the plurality of secondary nodes once all of the plurality of secondary nodes have acknowledged the update, wherein

the clearing is performed in response to receiving the notification.

2. The method of claim 1 wherein

the clearing the update from the respective log comprises updating a start-of-log pointer in the respective log.

3. The method of claim 1 wherein

the clearing the update from the respective log comprises updating a pointer to a location in the respective log, wherein

the pointer points to the location if the location contains a next update to clear.

4. The method of claim 1 further comprising:

determining that a location of a next update in a first respective log of updates to a first respective copy of the data at a first secondary node of the secondary nodes differs from a corresponding location of the next update in a second respective log of updates to a second respective copy of the data at a second secondary node of the secondary nodes; and

identifying a set of updates in the first respective log, wherein each update of the set of updates is not in the second respective log; and

synchronizing the first respective copy and the second respective copy by applying the set of updates to the second respective copy.

5. The method of claim 4 wherein

the determining occurs when a primary node maintaining the data fails.

6. The method of claim 1 further comprising:

setting a sent indicator for the update for one of the plurality of secondary nodes when the update is sent to the one secondary node.

7. The method of claim 6 further comprising:

setting a received indicator for the update for the one secondary node when an acknowledgement of the update is received from the one secondary node.

8. The method of claim 7 wherein

the sending the notification to each of the plurality of secondary nodes comprises determining that a respective sent indicator and a respective received indicator for the update are set for each of the plurality of secondary nodes.

9. The method of claim 1, further comprising:

in response to the identifying, incrementing a regional counter stored on the primary node by a number of secondary nodes to which the update is sent, wherein

the regional counter is a number of secondary nodes from which an acknowledgement to the update is to be received;

in response to receiving an acknowledgement from a secondary node among the plurality of secondary nodes to which the update is sent, decrementing the regional counter; and

in response to the regional counter reaching a value prior to the incrementing, determining that each of the plurality of secondary nodes has acknowledged the update.

10. A computer-readable storage medium having a plurality of instructions embodied therein, wherein the plurality of instructions are executable for:

identifying instructions to identify a plurality of secondary nodes;

sending an update from a primary node to the plurality of secondary nodes, wherein

the update identifies at least one secondary node of the plurality of secondary nodes to which the update will be sent;

in response to receiving the update from the primary node, causing the at least one secondary node to insert the update in a respective log of updates, wherein

each of the respective log of updates corresponds to a respective copy of the data, and

the inserting the update in the respective log of updates is performed at the secondary node;

in response to inserting the update in the respective log of updates, causing the update to be copied from the respective log of updates to a storage area at the least one secondary node;

sending an acknowledgement from the at least one secondary node to the primary node, wherein

the acknowledgement indicates that the update has been received at the least one secondary node;

determining that all of the plurality of secondary nodes have acknowledged the update; and

in response to the determining, causing each secondary node of the at least one secondary node to clear the update from the respective log of updates by sending instructions to send a notification to each of the plurality of secondary nodes once all of the plurality of secondary nodes have acknowledged the update, wherein

the clearing is performed in response to receiving the notification.

11. The computer-readable storage medium of claim 10 wherein the clearing instructions further comprise

updating instructions to update a start-of-log pointer in the respective log.

12. The computer-readable storage medium of claim 10 wherein the clearing instructions further comprise

updating instructions to update a pointer to a location in the respective log, wherein

the pointer points to the location if the location contains a next update to clear.

13. The computer-readable storage medium of claim 10 further comprising:

determining instructions to determine that a location of a next update in a first respective log of updates to a first respective copy of the data at a first secondary node of the secondary nodes differs from a corresponding location of the next update in a second respective log of updates to a second respective copy of the data at a second secondary node of the secondary nodes; and

second identifying instructions to identify a set of updates in the first respective log, wherein

each update of the set of updates is not in the second respective log; and

synchronizing instructions to synchronize the first respective copy and the second respective copy by applying the set of updates to the second respective copy.

14. The computer-readable storage medium of claim 10, further comprising:

incrementing instructions to increment a regional counter stored on the primary node by a number of secondary nodes to which the update is sent, in response to the identifying, wherein

the regional counter is a number of secondary nodes from which an acknowledgement to the update has not been received;

decrementing instructions to decrement the regional counter, in response to receiving an acknowledgement from a secondary node among the plurality of secondary nodes to which the update is sent; and

determining instructions to determine that each of the plurality of secondary nodes has acknowledged the update.

15. A computer system comprising:

a processor for executing instructions, and

a memory to store the instructions, wherein the instructions comprise

identifying instructions to identify a plurality of secondary nodes to which an update to data is sent from a primary node, wherein

the update identifies at least one secondary node of the plurality of secondary nodes to which the update will be sent;

in response to receiving the update from the primary node, causing the at least one secondary node to insert the update in a respective log of updates to a respective copy of the data, wherein

each of the respective log of updates corresponds to a respective copy of the data, and

the inserting the update in the respective log of updates is performed at the secondary node;

in response to inserting the update in the respective log of updates, causing the update to be copied from the respective log of updates to a storage area at the least one secondary node;

sending an acknowledgement from the at least one secondary node to the primary node, wherein

the acknowledgement indicates that the update has been received at the least one secondary node;

determining that all of the plurality of secondary nodes have acknowledged the update; and

in response to the determining, causing each secondary node of the at least one secondary node to clear the update from the respective log of updates by sending instructions to send a notification to each of the plurality of secondary nodes when all of the plurality of secondary nodes have acknowledged the update, wherein

the clearing is performed in response to receiving the notification.

16. The computer system of claim 15 wherein the instructions further comprise

determining instructions to determine that a location of a next update in a first respective log of updates to a first respective copy of the data at a first secondary node of the secondary nodes differs from a corresponding location of the next update in a second respective log of updates to a second respective copy of the data at a second secondary node of the secondary nodes;

and

second identifying instructions to identify a set of updates in the first respective log, wherein

each update of the set of updates is not in the second respective log; and

synchronizing instructions to synchronize the first respective copy and the second respective copy by applying the set of updates to the second respective copy.

17. The computer system of claim 15, wherein the instructions further comprise:

incrementing instructions to increment a regional counter stored on the primary node by a number of secondary nodes to which the update is sent, in response to the identifying, wherein

the regional counter is a number of secondary nodes from which an acknowledgement to the update has not been received;

decrementing instructions to decrement the regional counter, in response to receiving an acknowledgement from a secondary node among the plurality of secondary nodes to which the update is sent; and

determining instructions to determine that each of the plurality of secondary nodes has acknowledged the update.