Active failure detection

• US 5,983,371 A
• Filed: 07/11/1997
• Issued: 11/09/1999
• Est. Priority Date: 07/11/1997
• Status: Expired due to Term

• Current Assignee: Stratus Technologies Bermuda Ltd. (Stratus Technologies Bermuda Holdings Ltd.)
• Original Assignee: Marathon Technologies Corporation (Stratus Technologies Bermuda Holdings Ltd.)
• Inventors: Lord, Christopher C., Schwartz, David B.
• Primary Examiner(s): Beausoliel, Jr., Robert W.
• Assistant Examiner(s): Elisca, Pierre E.
• Application Number:
  US08/891,539
• Time in Patent Office: 851 Days
• Field of Search: 395/185.08, 395/185.09, 395/183.19, 395/182.1, 395/182.07, 395/182.08, 395/182.09, 395/845, 395/846, 395/878, 395/559, 395/553, 364/131, 364/143, 371/5.4, 371/42, 371/47.1
• US Class Current: 714/55
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/1633 using mutual exchange of the output between the redundant processing components

Abstract

Failures in a fault-tolerant computer system which includes two or more input/output processors connected to a data communication system are detected by monitoring data communication. The computer system is able to detect failures associated with a primary input/output processor, as well as with a standby input/output processors, and is also able to discriminate between failures of the input/output processors and communication failures in the data communication network itself. In addition to using heartbeat-like transmissions, various other categories of data communication are also used to detect failures. The system is able to detect failures when the input/output processors are on a common network segment, allowing the processors to monitor identical data traffic, as well as when the processors are on different segments where, as a result of filtering behavior of network elements such as active hubs, the processors may not be able to monitor identical data traffic.

Claims

1. A method for detecting a failure in a fault-tolerant computer system that includes a first input/output processor and a second input/output processor coupled to a data communication system, the method comprising the steps of:

processing a first data communication at the first input/output processor;

applying a timing criterion to a category of the first data communication processed by the first input/output processor to produce a first timing result;

processing a second data communication at the second input/output processor;

applying the timing criterion to the category of the second data communication processed by the second input/output processor to produce a second timing result;

determining a relationship between the first timing result and the second timing result; and

detecting whether a failure has occurred based on the determined relationship.

2. The method of claim 1 wherein the step of detecting whether the failure has occurred includes determining that a failure has occurred when a difference between the timing results exceeds a threshold value.

3. The method of claim 1 wherein the timing criterion is a time of last transmission or reception.

4. The method of claim 1 wherein the category of data communications includes messages originating from the first input/output processor.

5. The method of claim 4 wherein the category of data communications includes messages originating from the first input/output processor and directed to an address to which the second input/output processor is normally responsive.

6. The method of claim 4 wherein the category of data communications includes messages sent from the first input/output processor and directed through the data communication system to the second input/output processor.

7. The method of claim 1 wherein the category of data communications includes messages originating outside the computer system.

8. The method of claim 7 wherein the messages originate at a second computer coupled to the data communication system.

9. The method of claim 7 wherein the category of data communications includes messages originating outside the computer system and addressed to a group of systems of which the computer system is a member.

10. The method of claim 1 wherein the category of data communications includes messages originating from a third element of the computer system in data communication with the input/output processors.

11. The method of claim 1 further comprising the step of sending the first timing result from the first input/output processor to the second input/output processor wherein:

the step of applying the timing criterion to the category of the first data communication processed by the first input/output processor includes applying the timing criterion to the category of the first data communication processed by the first input/output processor at the first input/output processor;

the step of applying the timing criterion to the category of the second data communication processed by the second input/output processor includes applying the timing criterion to the category of the second data communication processed by the second input/output processor at the second input/output processor; and

the step of determining a relationship between the timing results includes determining a difference between the timing results at the second input/output processor.

12. The method of claim 11 wherein the first timing result is sent over a dedicated communication channel between the first input/output processor and the second input/output processor.

13. The method of claim 1 wherein the step of applying the timing criterion to the category of the first data communication processed by the first input/output processor further includes applying a plurality of timing criteria to a corresponding plurality of categories of the first data communication processed by the first input/output processor to produce a first plurality of timing results;

the step of applying the timing criterion to the category of the second data communication processed by the second input/output processor further includes applying the plurality of timing criteria to the corresponding plurality of categories of the second data communication processed by the second input/output processor to produce a second plurality of timing results; and

the step of determining a relationship between the timing results further includes determining relationships between corresponding ones of the first plurality of timing results and the second plurality of timing results.

14. A fault-tolerant computer system coupled to a data communication system comprising:

a first input/output processor configured to process a category of data communications and to apply a timing criterion to the category of data communications to produce a first timing result; and

a second input/output processor configured to process the category of data communications and to apply a timing criterion to the category of data communications to produce a second timing result;

wherein the computer system is configured to determine a relationship between the timing results and to determine whether a failure has occurred based on the relationship.

15. The system of claim 14 wherein the timing criterion is a time of last transmission or reception.

16. The system of claim 14 wherein the category of data communications includes messages originating from the first input/output processor.

17. The system of claim 16 wherein the category of data communications includes messages sent from the first input/output processor and directed through the data communication system to the second input/output processor.

18. The system of claim 16 wherein the category of data communications includes messages originating from the first input/output processor and directed to an address to which the second input/output processor is normally responsive.

19. The system of claim 14 wherein the category of data communications includes messages originating outside the fault-tolerant system.

20. The system of claim 19 wherein the messages originate at a second computer coupled to the data communication system.

21. The system of claim 19 wherein the category of data communications includes messages originating outside the fault-tolerant system and addressed to a group of systems of which the fault-tolerant system is a member.

22. The system of claim 14 further comprising a third element of the computer system in data communication with the input/output processors, and wherein the category of data communications includes messages originating from said third element.

23. The system of claim 14 further comprising:

a dedicated communication channel coupling the first input/output processor and the second input/output processor, the communication channel being configured to send the first timing result from the first input/output processor to the second input/output processor;

wherein the second input/output processor is configured to determine a difference between the timing results and to detect whether the failure has occurred when the difference exceeds a threshold.

24. The system of claim 14 wherein:

the first input/output processor is further configured to apply a plurality of timing criteria to a corresponding plurality of categories of data communications processed by the first input/output processor to produce a first plurality of timing results;

the second input/output processor is further configured to apply the plurality of timing criteria to the corresponding plurality of categories of data communications processed by the second input/output processor to produce a second plurality of timing results; and

the computer system is further configured to determine a relationships between corresponding ones of the first and second plurality of timing results and to determine whether the failure has occurred based on the determined relationships.