Fault resilient/fault tolerant computing

• US 20020026604A1
• Filed: 08/10/2001
• Published: 02/28/2002
• Est. Priority Date: 11/14/1997
• Status: Active Grant

• Current Assignee: Marathon Technologies Corporation (Stratus Technologies Bermuda Holdings Ltd.)
• Original Assignee: Marathon Technologies Corporation (Stratus Technologies Bermuda Holdings Ltd.)
• Inventors: Bissett, Thomas D., Muench, Erik, Leveille, Paul A.
• Granted Patent: US 6,473,869 B2
• Time in Patent Office: Days
• US Class Current: 714/12
• CPC Class Codes: G06F 11/1633 using mutual exchange of the output between the redundant processing components
• CPC Class Codes: G06F 11/1641 where the comparison is not performed by the redundant processing components
• CPC Class Codes: G06F 11/165 with continued operation after detection of the error
• CPC Class Codes: G06F 11/1658 Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
• CPC Class Codes: G06F 11/1679 at clock signal level
• CPC Class Codes: G06F 11/1687 at event level, e.g. by interrupt or result of polling
• CPC Class Codes: G06F 11/1691 using a quantum

Abstract

A fault tolerant/fault resilient computer system includes at least two compute elements connected to at least one controller. Each compute element has clocks that operate asynchronously to clocks of the other compute elements. The compute elements operate in a first mode in which the compute elements each execute a first stream of instructions in emulated clock lockstep, and in a second mode in which the compute elements each execute a second stream of instructions in instruction lockstep. Each compute element may be a multi-processor compute element.

Claims

1. A fault tolerant/fault resilient computer system, comprising:

at least two compute elements; and

at least one controller connected to the at least two compute elements;

wherein;

each of the compute elements has clocks that operate asynchronously to clocks of the other compute elements,

the compute elements operate in a first mode in which the compute elements each execute a first stream of instructions in emulated clock lockstep, and

the compute elements operate in a second mode in which the compute elements each execute a second stream of instructions in instruction lockstep.

2. The computer system of claim 1, wherein the at least two compute elements each comprise a multi-processor compute element.

3. The computer system of claim 2, wherein the at least two compute elements each comprise a symmetric multi-processor (SMP) compute element.

4. The computer system of claim 2, wherein each compute element is implemented using an industry standard motherboard.

5. The computer system of claim 2, wherein the first stream of instructions comprises operating system and application software and the second stream of instructions comprises lockstep control software.

6. The computer system of claim 5, wherein the operating system and application software comprise unmodified software configured for use with computer systems that are not fault tolerant.

7. The computer system of claim 2, wherein the system is configured to deactivate all but one of the processors of each compute element when the compute elements are operating in the second mode.

8. The computer system of claim 1, wherein each compute element comprises a processor, memory, and a connection to the controller.

9. The computer system of claim 8, where each compute element is configured so that refresh operations associated with the memory are synchronized with execution of operations by the processor.

10. The computer system of claim 8, wherein the system is configured to initiate DMA transfers to the memory when the compute elements are operating in the second mode and to execute the initiated DMA transfers when the compute elements are operating in the first mode.

11. The computer system of claim 8, wherein the system is configured to synchronize compute elements by:

copying contents of the memory of a first compute element to the memory of a second compute element; and

resetting the processors of the first and second compute elements without affecting the memories of the compute elements.

12. The computer system of claim 1, wherein each compute element is configured to transition from the first mode of operation to the second mode of operation in response to an interrupt.

13. The computer system of claim 12, wherein the interrupt comprises a performance counter interrupt generated by the compute element after the occurrence of a fixed number of clock cycles.

14. The computer system of claim 13, wherein the interrupt comprises a performance counter interrupt generated by the compute element after the occurrence of a fixed number of processor clock cycles.

15. The computer system of claim 13, wherein the interrupt comprises a performance counter interrupt generated by the compute element after the occurrence of a fixed number of bus clock cycles.

16. The computer system of claim 12, wherein the interrupt comprises an interrupt generated by the compute element after the execution of a fixed number of instructions.

17. The computer system of claim 12, wherein the at least two compute elements each comprise a multi-processor compute element having a primary processor and one or more secondary processors, and wherein the primary processor is configured to halt operation of the secondary processors in response to the interrupt.

18. The computer system of claim 1, wherein each compute element is configured to generate an interrupt during transition from the second mode of operation to the first mode of operation, the interrupt serving to align the processing by the compute element with a clocking structure of the compute element.

19. The computer system of claim 18, wherein the interrupt is synchronized with a clock having the lowest frequencies of the clocking structure.

20. The computer system of claim 1, wherein the system is configured to redirect I/O operations by the compute elements to the controller.

21. The computer system of claim 1, further comprising a second controller connected to the first controller and to the at least two compute elements.

22. The computer system of claim 21, wherein the first controller and a first one of the compute elements are located in a first location and the second controller and a second one of the compute elements are located in a second location, and further comprising a communications link connecting the first controller to the second controller, the first controller to the second one of the compute elements, and the second controller to the first one of the compute elements.

23. The computer system of claim 22, wherein the first location is spaced from the second location by more than 5 meters.

24. The computer system of claim 23, wherein the first location is spaced from the second location by more than 100 meters.

26. The method of claim 25, wherein each compute element comprises a multi-processor compute element.

27. The method of claim 26, wherein the first stream of instructions comprises operating system and application software and the second stream of instructions comprises lockstep control software.

28. The method of claim 26, further comprising deactivating all but one of the processors of each compute element when operating the compute elements in the second mode.

29. The method of claim 25, wherein each compute element comprises a processor, memory, and a connection to the controller.

30. The method of claim 29, further comprising synchronizing refresh operations associated with the memory with execution of operations by the processor.

31. The method of claim 29, further comprising initiating DMA transfers to the memory when operating the compute elements in the second mode and executing the initiated DMA transfers when operating the compute elements in the first mode.

32. The method of claim 29, further comprising synchronizing the compute elements by:

copying contents of the memory of a first compute element to the memory of a second compute element; and

resetting the processors of the first and second compute elements without affecting the memories of the compute elements.

33. The method of claim 25, further comprising transitioning from operating in the first mode to operating in the second mode in response to an interrupt.

34. The method of claim 33, wherein the at least two compute elements each include a multi-processor compute element having a primary processor and one or more secondary processors, the method further comprising halting operation of the secondary processors in response to the interrupt.

35. The method of claim 25, further comprising redirecting I/O operations by the compute elements to the controller.

25. A method of operating a fault tolerant/fault resilient computer system having at least two compute elements connected to at least one controller and having clocks that operate asynchronously to clocks of the other compute elements, the method comprising:

operating the compute elements in a first mode in which the compute elements each execute a first stream of instructions in emulated clock lockstep, and

operating the compute elements in a second mode in which the compute elements each execute a second stream of instructions in instruction lockstep.