Method to recover from node failure/recovery incidents in distributed sytems in which notification does not occur

• US 20040146070A1
• Filed: 01/27/2003
• Published: 07/29/2004
• Est. Priority Date: 01/27/2003
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Blackmore, Robert S., Kim, Chulho, Sivaram, Rajeev, Epter, Scott D., Chen, Amy X.
• Granted Patent: US 7,379,444 B2
• Time in Patent Office: Days
• US Class Current: 370/503
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

Epoch numbers are maintained in a pair wise fashion at a plurality of communication endpoints to provide communication consistency and recovery from a range of failure conditions including total or partial node failure and subsequent recovery. Once an epoch state inconsistency is recognized, negotiation procedures provide an effective mechanism to reestablish valid communication links without the need to employ global variables which inherently possess greater transmission and overhead requirements needed to maintain communications. Renegotiation of recognizably valid epoch numbers occurs on a pair wise basis.

Claims

1. A method, for establishing communications in a distributed data processing system, said method comprising:

maintaining, at each one of a plurality of communication endpoints in said distributed data processing system, pair wise epoch number indications relative to other ones of said plurality of communication endpoints and negotiating message transmission between pairs of said endpoints based on said pair wise epoch number indications, whereby the need for global status maintenance and transmission is eliminated.

2. A method for establishing communication in a distributed data processing system, said method comprising the steps of:

determining the existence of a state discrepancy between a communication source and a communication destination;

determining that said discrepancy indicates that neither said source nor said destination has undergone a system reset and that a failure at said source has occurred; and

negotiating between said source and said destination to establish a new state at said destination consistent with the state at said source so as to permit said source to exit from a non-ready state with respect to said destination.

3. The method of claim 2 in which said negotiation comprising the steps of:

said source incrementing its epoch number with respect to said destination and sending a ready request message to said destination;

said destination receiving said ready request, increments its epoch number with respect to said source and sending a ready acknowledgment message to said source;

said source receiving said ready acknowledgment message, setting its own ready state with respect to said destination and transmitting a message to said destination; and

said destination receiving said message with a matching epoch number and sets its ready state with respect to said source.

5. The method of claim 2 in which said negotiation comprising the steps of:

receiving at said destination a message with a non-matching epoch number, dropping said message and sending a not-ready acknowledgment message to said source along with the same epoch number sent by said source;

receiving at said source said not-ready acknowledgment message, setting said source into a non-ready state with respect to said destination, incrementing the received epoch number and sending a ready request message to said destination;

receiving at said destination said ready request message from said source, setting its epoch number with respect to said source to the received value and sending a ready acknowledgment message to said source;

receiving at said source said ready acknowledgment message, setting its ready state with respect to said destination and transmitting a message to said destination; and

receiving said message at said destination with a matching epoch number and setting its ready state with respect to said source.

4. A method for establishing communication in a distributed data processing system, said method comprising the steps of:

determining the existence of a state discrepancy between a communication source and a communication destination;

determining that said discrepancy indicates that neither said source nor said destination has undergone a system reset and that a failure at said destination has occurred; and

negotiating between said source and said destination to establish a new state at said destination consistent with the state at said source so as to permit said destination to exit from a non-ready state with respect to said source.

6. A method for establishing communication in a distributed data processing system, said method comprising the steps of:

determining the existence of a state discrepancy between a communication source and a communication destination;

determining that said discrepancy indicates that said source has undergone a system reset; and

negotiating between said source and said destination to establish a new state at said destination consistent with the state at said source so as to permit said destination to exit from a non-ready state with respect to said source.

7. The method of claim 6 in which said negotiation comprising the steps of:

said source setting its epoch number so as to indicate a system reset at said source and sending said epoch number to said destination with a ready request message;

receiving said epoch number at said destination and maintaining a count of such received epoch numbers until a threshold is reached;

once said threshold is reached, incrementing said epoch number and sending a ready acknowledgment message to said source along with said updated epoch number;

receiving at said source said ready acknowledgment message and said updated epoch number, setting its epoch number with respect to said destination to said updated epoch number, setting its ready state with respect to said destination and transmitting a message to said destination; and

receiving said message at said destination with a matching epoch number and setting its ready state with respect to said source.

8. A method for establishing communication in a distributed data processing system, said method comprising the steps of:

determining the existence of a state discrepancy between a communication source and a communication destination;

determining that said discrepancy indicates that said destination has undergone a system reset; and

negotiating between said source and said destination to establish a new state at said destination consistent with the state at said source so as to permit said destination to exit from a non-ready state with respect to said source.

9. The method of claim 8 in which said negotiation comprising the steps of:

said destination setting its epoch number so as to indicate a system reset at said destination and sending said epoch number to said source with a not ready acknowledgment message;

receiving at said source said epoch number and said not ready acknowledgment message and maintaining a count of such received epoch numbers until a threshold is reached;

once said threshold is reached, incrementing said epoch number at said source and sending a ready request message to said destination along with said incremented epoch number;

receiving at said destination said ready request message and said incremented epoch number, setting its epoch number with respect to said destination to said incremented epoch number, and sending a ready acknowledgment message to said source;

receiving at said source said ready acknowledgment message, setting said ready state with respect to said destination and transmitting a message to said destination with said incremented epoch number; and

receiving said message at said destination with a matching epoch number and setting its ready state with respect to said source.

10. A computer readable medium having computer executable instructions causing a computer to maintain at each one of a plurality of communication endpoints in a distributed data processing system, pair wise epoch number indications relative to other ones of said plurality of communication endpoints and negotiate message transmission between pairs of said endpoints based on said pair wise epoch number indications.

11. A distributed data processing system having a plurality of nodes containing executable instructions, in memory locations within the nodes of said distributed data processing system, for causing nodes in said distributed data processing system to maintain at each one of a plurality of said nodes in said distributed data processing system, pair wise epoch number indications relative to other ones of said plurality of nodes in said distributed data processing system and for causing said nodes in said distributed data processing system to negotiate message transmission between pairs of said nodes in said distributed data processing system based on said pair wise epoch number indications.