Fault tolerant computer system

• US 20030182594A1
• Filed: 03/14/2003
• Published: 09/25/2003
• Est. Priority Date: 03/19/2002
• Status: Active Grant

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Watkins, John E., Garnett, Paul J., Rowlinson, Stephen
• Granted Patent: US 7,124,319 B2
• Time in Patent Office: Days
• US Class Current: 714/11
• CPC Class Codes: H04L 1/22 using redundant apparatus to increase reliability

Abstract

A fault tolerant computing system is provided comprising two or more processing sets that operate in synchronism with one another. The two processing sets are joined by a bridge, and there is a communications link for each processing set for transmitting data from the processing set to the bridge. Data transmissions are initiated in synchronism with one another from the respective processing sets to the bridge but are then subject to variable delay over the communications link. Accordingly, a buffer is included in the bridge for storing the data transmissions received from the processing sets for long enough to compensate for the variable delay. The data transmissions can then be fed out from the buffer to a comparator that verifies that the data transmissions received from the two or more processing sets properly match each other. Likewise, a buffer is included in each processing set for storing the data transmissions received from the bridge for long enough to compensate for the variable delay. Control logic in each processing set can then apply the data transmissions to the respective processing set at a predetermined time.

Claims

1. A fault tolerant computing system comprising:

two or more processing sets operating in synchronism with one another;

a bridge;

a communications link between each processing set and the bridge, wherein data transmissions from the processing sets to the bridge are initiated in synchronism with one another but may then be subject to delay over the respective communications links, and wherein the delay may be variable between different ones of said communications links;

a buffer included in said bridge for storing the data transmissions received from the processing sets for long enough to compensate for said variable delay; and

a comparator for determining whether or not the data transmissions received from the two or more processing sets into said buffer match each other.

2. The fault tolerant computing system of claim 1, wherein said buffer comprises a FIFO queue associated with each link to a processing set.

3. The fault tolerant computing system of claim 2, further comprising control logic for synchronising data within each FIFO queue for output to said comparator.

4. The fault tolerant computing system of claim 1, wherein data transmissions between the processing sets and the bridge are structured into packets.

5. The fault tolerant computing system of claim 4, wherein said packets include an error correcting code (ECC).

6. The fault tolerant computing system of claim 4, wherein a packet contains a sending time, and the packet is transferred from said buffer to the comparator within a set time period after said sending time.

7. The fault tolerant computing system of claim 6, wherein the bridge and the processing sets share a global clock signal.

8. The fault tolerant computing system of claim 4, wherein said bridge determines a receipt time for a packet based on the earliest time it is received from one of said processing sets, and the packet is transferred from said buffer to the comparator within a set time period after said receipt time.

9. The fault tolerant computing system of claim 6, further comprising an error detector for flagging if a given packet has not been received from all the processing sets before expiry of said set time period.

10. The fault tolerant computing system of claim 6, wherein a packet contains a value for said set time period.

11. The fault tolerant computing system of claim 6, wherein said set time period is updated according to current delays on the communications links.

12. The fault tolerant computing system of claim 11, further comprising a monitor for assessing current delays on the communications links.

13. The fault tolerant computing system of claim 6, wherein a packet is transferred from said buffer to the comparator as soon as it has been received from all the processing sets, providing this is within said set time period.

14. The fault tolerant computing system of claim 4, wherein a packet contains an identifier to allow corresponding packets from different processing sets to be determined.

15. The fault tolerant computing system of claim 14, wherein a packet contains a sequence number to allow lost packets to be detected.

16. The fault tolerant computing system of claim 4, further comprising a flow control mechanism to prevent overflow of said buffer.

17. The fault tolerant computing system of claim 1, further comprising a bus attached to the comparator, wherein matched data transmissions are passed from the comparator to the bus.

18. The fault tolerant computing system of claim 1, wherein data transmissions from the bridge to each of the processing sets are initiated in synchronism with one another, but may then be subject to delay over the respective communications links, and wherein the delay may be variable between different ones of said communications links, and said system further comprises:

a buffer included in each processing set for storing the data transmissions received from the bridge for long enough to compensate for the variable delay; and

control logic in each processing set for applying the data transmissions to the respective processing set at a predetermined time.

19. The fault tolerant computing system of claim 18, wherein each data transmission from the bridge to the processing sets includes a specified time value, and said predetermined time occurs a set time period after said specified time value.

20. A fault tolerant computing system comprising:

two or more processing means operating in synchronism with one another;

means for bridging the two or more processing means;

means for linking each of said two or more processing means with the bridging means, wherein data transmissions from the processing means to the bridging means are initiated in synchronism with one another but may then be subject to delay over the respective linking means, and wherein the delay may be variable between different ones of said linking means;

means included in said bridging means for buffering the data transmissions received from the two or more processing means for long enough to accommodate said variable delay; and

means for determining whether or not the data transmissions received from the two or more processing means match one another.

21. A bridge for use in a fault tolerant computing system comprising two or more processing sets operating in synchronism with one another, said bridge including:

at least one communications interface for receiving data transmissions from each of the two or more processing sets, wherein data transmissions from the processing sets to the bridge initiated in synchronism with one another may then be subject to a delay in transit, and wherein said delay may be variable according to the particular processing set from which a data transmission is received;

a buffer for storing the data transmissions received from the processing sets for long enough to compensate for said variable delay; and

a comparator for determining whether or not the data transmissions received from the two or more processing sets into said buffer match each other.

22. The bridge of claim 21, further comprising a bus interface, whereby matched data transmissions are passed from the comparator to the bus interface.

23. The bridge of claim 21, wherein the buffer comprises a FIFO queue for each processing set.

24. The bridge of claim 23, further including control logic for synchronising data within each FIFO queue for output to the comparator.

25. A fault tolerant computing system comprising:

two or more processing sets operating in synchronism with one another;

a bridge;

a communications link between each processing set and the bridge, wherein data transmissions are initiated from the bridge to the processing sets in synchronism with one another, but may then be subject to delay over the respective communications links, and wherein there may be a variable delay between different ones of said communications links;

a buffer included in each processing set for storing the data transmissions received from the bridge for long enough to compensate for said variable delay; and

control logic in each processing set for applying the data transmissions to the respective processing set at a predetermined time.

26. The fault tolerant computing system of claim 25, wherein the two or more processing sets and the bridge share a common clock signal.

27. The fault tolerant computing system of claim 26, wherein each data transmission includes a specified time value, and said predetermined time occurs a set time period after said specified time value.

28. The fault tolerant computing system of claim 27, wherein each data transmission includes a value corresponding to said set time period.

29. The fault tolerant computing system of claim 27, wherein said set time period is updated according to current delays on the communications links.

30. A fault tolerant computing system comprising

two or more processing means operating in synchronism with one another;

means for bridging the two or more processing means;

means for linking each processing means to the bridging means, wherein data transmissions are initiated from the bridging means to the processing means in synchronism with one another but may then be subject to delay over the respective linking means, and wherein the delay may be variable between different ones of said linking means;

means for buffering the data transmissions received at the bridging means from the processing means for long enough to accommodate said variable delay; and

means for determining whether or not the data transmissions received from the two or more processing means match one another.

31. A processing set for use in a fault tolerant computing system comprising two or more processing sets operating in synchronism with one another and a bridge, said processing set including:

a communications interface for receiving data transmissions from the bridge, wherein data transmissions initiated from the bridge to the two or more processing sets in synchronism with one another may then be subject to variable delay in transit;

a buffer for storing the data transmissions received from the bridge for long enough to compensate for said variable delay; and

control logic for applying the data transmissions at a predetermined time.

32. A fault tolerant computing system comprising:

two or more processing sets operating in synchronism with one another;

a bridge;

a communications link for each processing set for transmitting data between the processing set and the bridge, wherein data transmissions over the respective communications links are initiated in synchronism with one another but may then be subject to delay, and wherein the delay may be variable between different ones of said communications links;

a buffer included in said bridge for storing data transmissions received from the processing sets for long enough to compensate for said variable delay;

a comparator for determining whether or not the data transmissions received from the two or more processing sets into said buffer match each other.

a buffer included in each processing set for storing data transmissions received from the bridge for long enough to compensate for said variable delay; and

control logic in each processing set for applying the received data transmissions to the respective processing set at a predetermined time.

33. A method for operating a fault tolerant computing system comprising two or more processing sets operating in synchronism with one another, a bridge and a communications link connecting each processing set to the bridge, said method comprising:

transmitting data from the processing sets to the bridge, wherein data transmissions from the processing sets to the bridge are initiated in synchronism with one another, but may then be subject to delay over the respective communications links, and wherein the delay may be variable between different ones of said communications links;

buffering the data transmissions received at the bridge from the processing sets for long enough to compensate for said variable delay; and

then

determining whether or not the data transmissions received from the two or more processing sets match one another.

34. The method of claim 33, wherein said bridge comprises a FIFO queue associated with each link to a processing set for performing said buffering.

35. The method of claim 34, wherein said buffering includes synchronising data within each FIFO queue.

36. The method of claim 33, wherein data transmissions between the processing sets and the bridge are structured into packets.

37. The method of claim 36, wherein said packets include an error correcting code (ECC).

38. The method of claim 36, wherein a packet contains a sending time, and said buffering includes transferring the packet for verification within a set time period after said sending time.

39. The method of claim 38, wherein the bridge and the processing sets share a global clock signal.

40. The method of claim 36, wherein said buffering includes determining a receipt time for a packet based on the earliest time it is received at the bridge from one of said processing sets, and transferring the packet for verification within a set time period after said receipt time.

41. The method of claim 38, further comprising flagging if a given packet has not been received from all the processing sets before expiry of said set time period.

42. The method of claim 38, wherein a packet contains a value for said set time period.

43. The method of claim 38, further comprising updating said set time period according to current delays on the communications links.

44. The method of claim 43, further comprising monitoring current delays on the communications links.

45. The method of claim 38, wherein a packet is transferred for verification as soon as it has been received from all the processing sets, providing this is within said set time period.

46. The method of claim 36, wherein a packet contains an identifier to allow corresponding packets from different processing sets to be determined.

47. The method of claim 36, wherein a packet contains a sequence number to allow lost packets to be detected.

48. The method of claim 36, further comprising providing a flow control mechanism to prevent overflow of said buffer.

49. The method of claim 33, wherein said fault tolerant system further comprises a bus, whereby matched data transmissions are passed out onto the bus.

50. The method of claim 33, further comprising:

transmitting data from the bridge to each processing set, wherein data transmissions are initiated from the bridge to the processing sets in synchronism with one another, but may then be subject to delay over the respective communications links, and wherein there may be a variable delay between different ones of said communications links;

buffering the data transmissions received at the processing sets from the bridge for long enough to compensate for said variable delay; and

applying the data transmissions to the respective processing set at a predetermined time.

51. The method of claim 50, wherein each data transmission includes a specified time value, and said predetermined time occurs a set time period after said specified time value.

52. A method for operating a fault tolerant computing system comprising two or more processing sets operating in synchronism with one another, a bridge and a communications link connecting each processing set to the bridge, said method comprising:

transmitting data from the bridge to each processing set, wherein the data transmissions are initiated from the bridge to the processing sets in synchronism with one another, but may then be subject to delay over the respective communications links, and wherein the delay may be variable between different ones of said communications links;

buffering the data transmissions received at the processing sets from the bridge for long enough to compensate for said variable delay; and

applying the data transmissions to the respective processing set at a predetermined time.

53. The method of claim 52, wherein the two or more processing sets and the bridge share a common clock signal.

54. The method of claim 53, wherein each data transmission includes a specified time value, and said predetermined time occurs a set time period after said specified time value.

55. The method of claim 54, wherein each data transmission includes a value corresponding to said set time period.

56. The method of claim 54, further comprising updating said set time period according to current delays on the communications links.

57. The method of claim 56, further comprising monitoring current delays on the communications links.

58. A method for operating a bridge for use in a fault tolerant computing system comprising two or more processing sets operating in synchronism with one another, a bridge and a communications link connecting each processing set to the bridge, said method comprising:

receiving data transmissions at the bridge from the processing sets, wherein data transmissions from the processing sets to the bridge initiated in synchronism with one another may be subject to a delay over the respective communications links, and wherein the delay may be variable between different ones of said communications links;

buffering the received data transmissions for long enough to compensate for said variable delay; and

then

determining whether or not the received data transmissions match each other.

59. A method for operating a processing set for use in a fault tolerant computing system comprising two or more processing sets operating in synchronism with one another, a bridge and a communications link connecting each processing set to the bridge, said method comprising:

receiving data transmissions at a processing set from the bridge, wherein data transmissions initiated in synchronism with one another from the processing sets to the bridge may be subject to delay over the respective communications links, and wherein there may be a variable delay between different ones of said communications links;

buffering the received data transmissions for long enough to compensate for said variable delay; and

then

applying the received data transmissions to the processing set at a predetermined time.