Method for detecting the quick restart of liveness daemons in a distributed multinode data processing system

• US 7,203,748 B2
• Filed: 02/15/2002
• Issued: 04/10/2007
• Est. Priority Date: 02/15/2002
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Knop, Felipe, Hare, John R., Lin, Tseng-Hui
• Primary Examiner(s): Jaroenchonwanit; Bunjob
• Assistant Examiner(s): Refai; Ramsey
• Application Number:
  US10/078,076
• Publication Number:
  US 20030158921A1
• Time in Patent Office: 1,880 Days
• Field of Search: 709/223, 709/204, 709248-252, 709/208, 709/209, 709/224, 714/4
• US Class Current: 709/224
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

In distributed multinode data processing systems, mechanisms are employed to insure that the nodes are properly informed about the liveness of the other nodes in node groups in the network. In particular, the present invention employs group membership indicia as part of a mechanism for detecting that a node and/or its adapter have failed and have been recently restarted. Having detected this situation, group membership inconsistencies which it can engender are avoided.

Claims

1. A method for facilitating correct group membership by detecting the quick restart of liveness daemons in a distributed, multimode data processing system in which nodes communicate liveness indicia in the form of heartbeat signals via adapters coupled to each node, said method comprising:

experiencing at one node of a membership group a failure and a quick restart of the one node of a membership group;

wherein the failure and quick restart deletes locally stored membership group information at said one node;

subsequent to said failure and quick restart of the one node of a membership group, receiving a heartbeat signal at the one node from at least one other node of the membership group;

responsive to receipt of the heartbeat signal at the one node, sending, from the one node to the at least one other node, a first message which includes an indication the quick restart at the one node; and

determining, at the at least one other node, from said indication of the quick restart in the first message and from locally stored membership group information indicating prior membership of the one node in the membership group, the occurrence of a quick restart at said one node, and responding thereto by sending a second message from the at least one other node to another node of the membership group which indicates that said one node is to be expelled from the membership group;

wherein the quick restart at the one node occurs prior to a detection of the failure by at least one other node and expulsion of the one node from the membership group due to the failure.

2. The method of claim 1 in which said second message is sent by the node that is next in line for receipt of heartbeat signals, with respect to the node that sent the first message.

3. The method of claim 1, wherein the membership group is an adapter membership group, and said indication of quick restart are selected from the group consisting of:

(1) an indication that the one node and the at least one other node are not both in the adapter membership group;

(2) an indication that the one node'"'"'s address is part of the adapter membership group according to said at least one other node; and

(3) an indication of difference in instantiation number for the one node'"'"'s adapter ID listed in the adapter membership group.

5. The method of claim 3, wherein the indication of the quick restart includes each of (1) an indication that the one node and the at least one other node are not in the adapter membership group;

(2) an indication that the one node'"'"'s address is part of the adapter membership group according to the at least one other node; and

(3) an indication of difference in instantiation number for the one node'"'"'s adapter ID listed in the adapter membership group.

6. The method of claim 1, wherein the first message comprises a PROCLAIM message.

7. The method of claim 1, wherein the heartbeat signal comprises a “

HEARTBEAT”

message, the first message comprises a “

NOT YOUR NEIGHBOR”

message, and the second message comprises a “

DEATH”

message, wherein the at least one other node forwards the “

DEATH”

message to a group leader node of the membership group.

4. A multimode data processing system comprising:

a plurality of data processing nodes connected in a network capable of transmitting messages between nodes;

storage means within said nodes containing program code for experiencing at one node of a membership group, a failure and a quick restart of the one node of a membership group;

wherein the failure and quick restart deletes locally stored membership group information at said one node;

subsequent to said failure and quick restart of the one node of a membership group, receiving a heartbeat signal at the one node from at least one other node of the membership group;

responsive to receipt of the heartbeat signal at the one node, sending, from the one node to the at least one other node a first message which includes an indication of the quick restart at the one node; and

determining, at the at least one other node, from said indication of the quick restart in the firs message and from locally stored membership group information indicating prior membership of the one node in the membership group, the occurrence of a quick restart at said one node, and responding thereto by sending a second message from the at least one other node to another node of the membership group which indicates that said one node is to be expelled from the membership group;

wherein the quick restart at the one node occurs prior to a detection of the failure by at least one other node and expulsion of the one node from the membership group due to the failure.

8. The multinode data processing system of claim 4, in which said second message is sent by the at least one other node that is the upstream neighbor, in terms of heartbeat passing signals, of the one node that sent the first message.

9. The multinode data processing system of claim 4, wherein the membership group is an adapter membership group, and said indication of quick restart are selected from the group consisting of (1) an indication that the one node and the at least one other node are not both in the adapter membership group;

(2) an indication that the one node'"'"'s address is part of the adapter membership group according to said at least one other node; and

(3) an indication of difference in instantiation number for the one node'"'"'s adapter ID listed in the adapter membership group.

10. The multinode data processing system of claim 9, wherein the quick restart indicia includes each of:

(1) an indication that the one node and the at least one other node are not in the adapter membership group;

(2) an indication that the one nodes address is part of the adapter membership group according to the at least one other node; and

(3) an indication of difference in instantiation number for the one node'"'"'s adapter ID listed in the adapter membership group.

11. The multinode data processing system of claim 4, wherein the first message comprises a PROCLAIM message.

12. The multinode data processing system of claim 4, wherein the heartbeat signal comprises a “

HEARTBEAT”

message, the first message comprises a “

NOT YOUR NEIGHBOR”

message, and the second message comprises a “

DEATH”

message, wherein the at least one other node forwards the “

DEATH”

message to at group leader node of the membership group.

18. The at least one program storage device of claim 12, wherein the heartbeat signal comprises a “

HEARTBEAT”

message, the first message comprises a “

NOT YOUR NEIGHBOR”

message, and the second message comprises a “

DEATH”

message, wherein the at least one other node forwards the “

DEATH”

message to a group leader node of the membership group.

13. At least one program storage device readable by at least one computer, tangibly embodying at least one program of instructions executable by the at least one computer to perform a method of facilitating correct group membership by detecting quick restart of liveness daemons in a distributed, multinode data processing system in which nodes communicate liveness indicia in the form of heartbeat signals via adapters coupled to each other, said method comprising:

experiencing at one node of a membership group, a failure and a quick restart of the one node of a membership group;

wherein the failure and quick restart deletes locally stored membership group information at said one node;

subsequent to said failure and quick restart of the one node of a membership group, receiving a heartbeat signal at the one node from at least one other node of the membership group;

responsive to receipt of the heartbeat signal at the one node, sending, from the one node to the at least one other node, a first message which includes an indication of the quick restart at the one node; and

determining at the at least one other node, from said indication of the quick restart in the first message and from locally stored membership group information indicating prior membership of the one node in the membership group, the occurrence of a quick restart at said one node, and responding thereto by sending a second message from the at least one other node to another node of the membership group which indicates that said one node is to be expelled from the membership group;

wherein the quick restart at the one node occurs prior to a detection of the failure by at least one other node and expulsion of the one node from the membership group due to the failure.

14. The at least one program storage device of claim 13, in which said second message is sent by the at least one other node that is the upstream neighbor, in terms of heartbeat passing signals, of the one node that sent the first message.

15. The at least one program storage device of claim 13, wherein the membership group is an adapter membership group, and said indication of quick restart are selected from the group consisting of:

(1) an indication that the one node and the at least one other node are not both in the adapter membership group;

(2) an indication that the one nodes address is part of the adapter membership group according to said at least one other node; and

(3) an indication of difference in instantiation number for the one node'"'"'s adapter ID listed in the adapter membership group.

16. The at least one program storage device of claim 15, wherein the indication of the quick restart each of (1) an indication that the one node and the at least one other node are not in the adapter membership group;

(2) an indication that the one node'"'"'s address is part of the adapter membership group according to the at least one other node; and

(3) an indication of difference in instantiation number for the one node'"'"'s adapter ID listed in the adapter membership group.

17. The at least one program storage device of claim 13, wherein the first message comprises a PROCLAIM message.