Node failure recovery in a hub and spoke data replication mechanism

• US 6,335,937 B1
• Filed: 09/24/1998
• Issued: 01/01/2002
• Est. Priority Date: 09/24/1998
• Status: Expired due to Fees

• Current Assignee: Cogent Incorporated (Cognent, Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Shaheen, Amal Ahmed, Chao, Ching-Yun, Hough, Roger Eldred
• Primary Examiner(s): Chin, Wellington
• Assistant Examiner(s): Pham, Brenda H.
• Application Number:
  US09/160,021
• Time in Patent Office: 1,195 Days
• Field of Search: 370/216, 370/242, 370/312, 370/349, 370/395, 370/394, 370/346, 370/449, 370/426, 455/419, 455/420
• US Class Current: 370/426
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A node failure recovery mechanism for use in a data replicating system in a distributed computer environment wherein a plurality of servers are configured about one or more central hubs in a hub and spoke arrangement. In each of a plurality of originating nodes, updates and associated origination sequence numbers are sent to the central hub. The hub sends updates and associated distribution sequence numbers to the plurality of originating nodes. The hub tracks acknowledgments sent by nodes for a destination sequence number acknowledged by all nodes. Upon failure of a node, a node failure recovery method may be used to enable a “buddy” node to help the failed node gain readmission to a distribution group.

Claims

1. A method for replicating data in a distributed system comprising a plurality of originating nodes associated with a central hub, wherein origination nodes send updates and associated origination sequence numbers to the central hub, comprising the steps of:

from the central hub, sending updates and associated distribution sequence numbers to a distribution group comprising a plurality of originating nodes;

in the central hub, tracking periodic acknowledgments sent by originating nodes, each acknowledgment identifying a last in-sequence distribution sequence number processed by a respective originating node; and

isolating a given originating node from the distribution group upon a given occurrence.

2. A method for replicating data in a distributed system comprising a plurality of originating nodes associated with a central hub, wherein origination nodes send updates and associated origination sequence numbers to the central hub, comprising the steps of:

from the central hub, sending updates and associated distribution sequence numbers to a distribution group comprising a plurality of originating nodes;

in the central hub, tracking periodic acknowledgments sent by originating nodes, each acknowledgment identifying a last in-sequence distribution sequence number processed by a respective originating node; and

isolating a given originating node from the distribution group upon a given occurrence wherein the given occurrence is that the given originating node'"'"'s acknowledgments to the central hub indicate that the given originating node cannot keep pace with a remainder of the plurality with respect to processing of the updates.

3. A method for replicating data in a distributed system comprising a plurality of originating nodes associated with a central hub, wherein origination nodes send updates and associated origination sequence numbers to the central hub, comprising the steps of:

from the central hub, sending updates and associated distribution sequence numbers to a distribution group comprising a plurality of originating nodes;

in the central hub, tracking periodic acknowledgments sent by originating nodes, each acknowledgment identifying a last in-sequence distribution sequence number processed by a respective originating node; and

isolating a given originating node from the distribution group upon a given occurrence

wherein the given occurrence is that the given originating node is not timely providing periodic acknowledgments.

4. A method for replicating data in a distributed system comprising a plurality of originating nodes associated with a central hub, wherein origination nodes send updates and associated origination sequence numbers to the central hub, comprising the steps of:

from the central hub, sending updates and associated distribution sequence numbers to a distribution group comprising a plurality of originating nodes;

in the central hub, tracking periodic acknowledgments sent by originating nodes, each acknowledgment identifying a last in-sequence distribution sequence number processed by a respective originating node;

isolating a given originating node from the distribution group upon a given occurrence; and

determining whether the given originating node is attempting to rejoin the distribution group.

5. The method as described in claim 4 wherein if the given originating node is attempting to rejoin the distribution group, initiating a node recovery protocol to enable the given originating node to be readmitted to the distribution group.

6. The method as described in claim 5 wherein the node recovery protocol includes the steps of:

associating the given originating node with a buddy node; and

transferring a current copy of the data set from the buddy node to the given originating node.

7. The method as described in claim 6 further including the step of admitting the buddy node to the distribution group if necessary prior to transferring the current copy of the data set to the given originating node.

8. The method as described in claim 6 further including the steps of:

having the buddy node provide the central hub with its last in-sequence distribution sequence number prior to the transferring step.

9. A method for replicating data in a distributed system comprising a plurality of originating nodes associated with a central hub, wherein origination nodes send updates and associated origination sequence numbers to the central hub, comprising the steps of:

from the central hub, sending updates and associated distribution sequence numbers to a distribution group comprising a plurality of originating nodes;

in the central hub, tracking periodic acknowledgments sent by originating nodes, each acknowledgment identifying a last in-sequence distribution sequence number processed by a respective originating node; and

isolating a given originating node from the distribution group upon a given occurrence

wherein the isolating step includes the step of having the central hub notify other nodes that the given originating node has been isolated from the distribution group.

10. A method for replicating data in a distributed system comprising a plurality of originating nodes associated with a central hub, wherein origination nodes send updates and associated origination sequence numbers to the central hub, comprising the steps of:

from the central hub, sending updates and associated distribution sequence numbers to a distribution group comprising a plurality of originating nodes;

in the central hub, tracking periodic acknowledgments sent by originating nodes, each acknowledgment identifying a last in-sequence distribution sequence number processed by a respective originating node;

isolating a given originating node from the distribution group if the given originating node'"'"'s acknowledgments to the central hub indicate that the given originating node cannot keep pace with a remainder of the distribution group with respect to processing of the updates; and

selectively initiating a node failure recovery protocol to enable the given originating node to rejoin the distribution group.

11. The method as described in claim 10 wherein the node recovery protocol includes the steps of:

associating the given originating node with a buddy node;

readmitting the given originating node to the distribution group; and

transferring a current copy of the data set from the buddy node to the given originating node.

12. The method as described in claim 11 further including the step of admitting the buddy node to the distribution group if necessary prior to readmitting the given originating node.

13. The method as described in claim 10 wherein the readmitting step includes the steps of:

having the buddy node provide the central hub its last in-sequence distribution sequence number prior to the transferring step.

14. The method as described in claim 10 wherein the isolating step includes the step of having the central hub notify at least one other node that the given originating node has been isolated from the distribution group.

15. A computer program product in a computer-readable medium for replicating data in a distributed system comprising a plurality of originating nodes associated with a central hub, wherein origination nodes send updates and associated origination sequence numbers to the central hub, comprising:

means operative in the central hub for sending updates and associated distribution sequence numbers to a distribution group comprising a plurality of originating nodes;

means operative in the central hub for tracking acknowledgments sent by originating nodes, each acknowledgment identifying a last in-sequence distribution sequence number processed by a respective originating node; and

;

means operative in the central hub for isolating a given originating node from a given distribution group.

16. A computer program product in a computer-readable medium for replicating data in a distributed system comprising a plurality of originating nodes associated with a central hub, wherein origination nodes send updates and associated origination sequence numbers to the central hub, comprising:

means operative in the central hub for sending updates and associated distribution sequence numbers to a distribution group comprising a plurality of originating nodes;

means operative in the central hub for tracking acknowledgments sent by originating nodes, each acknowledgment identifying a last in-sequence distribution sequence number processed by a respective originating node; and

means operative in the central hub for isolating a given originating node from a given distribution group wherein the isolating means isolates the given originating node from the distribution group if the given originating node'"'"'s acknowledgments to the central hub indicate that the given originating node cannot keep pace with the remainder of the distribution group with respect to processing of the updates.

17. The computer program product as described in claim 16 further including means operative in the central hub and responsive to isolation of the given originating node for notifying other nodes that the given originating node has been isolated from the distribution group.

18. A computer program product in a computer-readable medium for replicating data in a distributed system comprising a plurality of originating nodes associated with a central hub, wherein origination nodes send updates and associated origination sequence numbers to the central hub, comprising:

means operative in the central hub for sending updates and associated distribution sequence numbers to a distribution group comprising a plurality of originating nodes;

means operative in the central hub for tracking acknowledgments sent by originating nodes, each acknowledgment identifying a last in-sequence distribution sequence number processed by a respective originating node;

means operative in the central hub for isolating a given originating node from a given distribution group; and

means for readmitting the given originating node back into the distribution group.

19. The computer program product as described in claim 18 wherein the readmitting means includes:

means for associating a buddy node to the given originating node; and

means for controlling the buddy node to transfer a current copy of a data set to the given originating node.

20. A computer for controlling replication of data in a distributed system comprising a plurality of originating nodes configured as spokes with respect to the computer, wherein origination nodes send updates and associated origination sequence numbers to the computer, comprising:

a processor;

storage means;

first instruction means executed by the processor for sending updates and associated distribution sequence numbers to a distribution group comprising a plurality of originating nodes;

second instruction means executed by the processor for tracking acknowledgments sent by originating nodes, each acknowledgment identifying a last in-sequence distribution sequence number processed by a respective originating node; and

;

third instruction means executed by the processor for isolating a given originating node from a given distribution group.

21. A computer for controlling replication of data in a distributed system comprising a plurality of originating nodes configured as spokes with respect to the computer, wherein origination nodes send updates and associated origination sequence numbers to the computer, comprising:

a processor;

storage means;

first instruction means executed by the processor for sending updates and associated distribution sequence numbers to a distribution group comprising a plurality of originating nodes;

second instruction means executed by the processor for each tracking acknowledgments sent by originating nodes, each acknowledgment identifying a last in-sequence distribution sequence number processed by a respective originating node;

third instruction means executed by the processor for isolating a given originating node from a given distribution group; and

fourth instruction means executed by the processor for readmitting the given originating node back into the distribution group.

22. The computer as described in claim 21 wherein the readmitting means includes:

means for associating a buddy node to the given originating node; and

means for controlling the buddy node to transfer a current copy of a data set to the given originating node.