Transparent TCP connection failover

• US 20040268175A1
• Filed: 06/11/2003
• Published: 12/30/2004
• Est. Priority Date: 06/11/2003
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Eternal Systems, Inc.
• Inventors: Melliar-Smith, Peter M., Hortikar, Sanjay, Moser, Louise E., Koch, Ruppert
• Granted Patent: US 7,251,745 B2
• Time in Patent Office: Days
• US Class Current: 714/4.11
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1034 Reaction to server failures by a load balancer
• CPC Class Codes: H04L 69/16 Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
• CPC Class Codes: H04L 69/161 Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields
• CPC Class Codes: H04L 69/163 In-band adaptation of TCP data exchange; In-band control procedures
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

Methods of transparent connection failover allowing a remote computer (i.e., a client), to continue to use a network connection to communicate with one of at least two or more other computers (i.e., the backup servers) over a network, when one of the other computers (i.e., the primary server) fails. With the mechanisms of this invention, there is no need for the client to establish a new connection to a backup server when the primary server fails. The failover is preferably executed within a bridge layer between the TCP layer and the IP layer of the server'"'"'s TCP/IP stack. No modifications are required to the network infrastructure, the client'"'"'s TCP/IP stack, the client application or the server application. The methods support active or semi-active replication of the server application, and do not require rollback of the application during failover. The invention also provides mechanisms for bringing up new backup servers.

Claims

1. An apparatus for providing transparent connection failover, comprising:

a first server computer and one or more second server computers executing the same server application and configured for executing communication routines for communicating over a network; and

means for migrating the connection endpoint of the server, within said communication routines on said computer servers upon the failure of said first server computer which is replying to requests from a client, so that said one or more second server computers begins replying to requests from said client without said client changing the connection endpoint address of the server from said first server computer to said one or more second server computers.

2. An apparatus as recited in claim 1, wherein said means of migrating the connection endpoint maintains synchronization between said one or more second server computers and said first server computer.

3. An apparatus as recited in claim 2, wherein said synchronization is maintained between said one or more second server computers and said first server computer, prior to a failure of said first server computer, by executing on said one or more second server computers the same requests that are being executed on said first server computer.

4. An apparatus as recited in claim 1, wherein said communication routines are executing a connection-oriented communication protocol over an underlying network communication protocol that carries data over said network.

5. An apparatus as recited in claim 4, wherein said means of migrating connection endpoint within said communication routines comprises a bridge layer in said first server computer and in said one or more second server computers between said connection-oriented protocol and said underlying network communication protocol.

6. An apparatus as recited in claim 5, wherein said first server computer and said one or more second server computers comprise a primary server being addressed by said client and one or more backup servers being maintained in synchronization with said primary server while said primary server is operational.

7. An apparatus as recited in claim 6, wherein said bridge layer in said primary server comprises a primary server output queue and a backup server output queue maintained by bridge routines within said bridge layer.

8. An apparatus as recited in claim 7, wherein said connection-oriented protocol comprises the Transmission Control Protocol (TCP) and said network communication protocol comprises the Internet Protocol (IP).

9. An apparatus as recited in claim 8, wherein said bridge layer modifies the Transmission Control Protocol (TCP) sequence numbers in the TCP segments so as to maintain synchronization between said first server computer and said one or more second server computers.

10. An apparatus for providing fault tolerance and transparent connection failover, comprising:

a computer server configured for executing communication protocols with client computers over a network; and

programming within said computer server for executing said communication protocols for

communicating requests from a client to a primary server over said network,

communicating requests from a client to at least one backup server over said network, where said backup server runs the same application program and performs the same operations in the application program as said primary server,

communicating responses from said primary server to said client, and

migrating the connection endpoint from said primary server to said backup server in response to the failure of said primary server;

wherein said backup server receives and processes said client requests without said client changing the endpoint address of the server from said primary server to said backup server, so that connection endpoint migration is transparent to said client.

11. An apparatus as recited in claim 10, further comprising programming configured for communicating client requests to said backup server to maintain synchronization of said backup server with said primary server.

12. An apparatus as recited in claim 11, wherein said primary server and said backup server undergo identical state transitions in the application, eliminating the need for transfer of application state to achieve transparent connection failover.

13. An apparatus as recited in claim 10, further comprising programming configured for transferring the application state and the state of communication protocols upon bringing up a new backup server replica, or returning a failed and repaired server replica to service.

14. An apparatus as recited in claim 10, wherein said communication protocols comprise a connection-oriented communication protocol executing over a network communication protocol.

15. An apparatus as recited in claim 14, wherein said connection-oriented protocol comprises the Transmission Control Protocol (TCP) and said network communication protocol comprises the Internet Protocol (IP).

16. An apparatus as recited in claim 14, wherein said programming is executed as bridge routines in a bridge sublayer, positioned between said connection-oriented communication protocol layer and said network communication protocol layer, said bridge routines configured for migrating the connection endpoint from said primary server to said backup server.

17. An apparatus as recited in claim 16, further comprising a primary server output queue and a backup server output queue maintained within said bridge layer for each fault-tolerant connection that is to be maintained between said client and said server.

18. An apparatus as recited in claim 16, further comprising programming within said bridge layer configured for adjusting the sequence numbers of outgoing segments, and acknowledgments for incoming segments, as received by said primary server from said connection-oriented communication protocol.

19. An apparatus as recited in claim 16, further comprising programming within said bridge layer configured for acknowledging a connection-oriented communication protocol segment from said client only after receiving an acknowledgment of that segment from said backup server.

20. An apparatus as recited in claim 16, further comprising programming within said bridge layer configured for maintaining a copy, on said backup server, of connection-oriented communication protocol segments, identical to the segments sent by said primary server, that have not been acknowledged by said client.

21. An apparatus as recited in claim 20, wherein said bridge routines are configured to remove a given connection-oriented communication protocol segment from the buffers maintained within said primary server and said backup server in response to acknowledgment by said client of said given connection-oriented protocol segment.

22. An apparatus as recited in claim 16:

wherein said bridge routines within said backup server are configured with programming for

replacing the original destination address of an outgoing segment, the client address, with the address of said primary server,

maintaining said client address from said outgoing message in an options field of said connection-oriented communication protocol, and

replacing the original destination address of an incoming segment, the primary server address, with the address of said backup server.

23. An apparatus as recited in claim 22, wherein said backup server is configured to operate on said network in a promiscuous mode in which it receives all datagrams from said client computer.

24. An apparatus as recited in claim 23:

wherein said connection-oriented communication protocol comprises the Transmission Control Protocol (TCP) and said network communication protocol comprises the Internet Protocol (IP);

wherein said backup server bridge discards the payload of any datagram not containing a Transmission Control Protocol (TCP) segment, or that is not addressed to said primary server;

wherein said backup server bridge replaces the original destination field with the address of said backup server within the Transmission Control Protocol (TCP) segment header and passes said TCP segment to said TCP layer of said backup server; and

wherein upon said backup server processing said Transmission Control Protocol (TCP) segment, said TCP layer of said backup server assumes that said client sent said segment directly to said backup server.

25. An apparatus as recited in claim 24:

wherein upon said primary server bridge receiving a Transmission Control Protocol (TCP) segment that said backup server sent, it matches the Transmission Control Protocol (TCP) segment'"'"'s payload data of said segment against the data in said primary server output queue and builds a new Transmission Control Protocol (TCP) segment containing the matching payload bytes; and

wherein the new Transmission Control Protocol (TCP) segment carries the address of said primary server in the source field and the address of said client in the destination field.

26. An apparatus as recited in claim 24:

wherein said backup server sends a Transmission Control Protocol (TCP) segment that does not contain the data bytes from the application program on said backup server;

wherein upon said primary server bridge receiving a Transmission Control Protocol (TCP) segment that said backup server sent, it matches the sequence numbers against the sequence numbers in said primary server output queue and builds a new Transmission Control Protocol (TCP) segment containing data bytes from said primary server output queue corresponding to the matching sequence numbers; and

wherein the new Transmission Control Protocol (TCP) segment carries the address of said primary server in the source field and the address of said client in the destination field.

27. In a server computer that executes a connection-oriented communication protocol over a network communication protocol for communicating requests and responses between a client computer and a primary server computer, the improvement comprising:

communication programming within said server computer for carrying out the operations of

communicating client requests to at least one backup server on said network, and

migrating the connection endpoint from said primary server to said backup server after the failure of said primary server in response to which said backup server receives and responds to said client requests while said client is still addressing said primary server over said network and is unaware of said server failure and said connection endpoint migration.

28. A method of ensuring transparent connection failover within a network, comprising:

executing communication routines on computers connected in said network so that computations can be executed on a first computer in response to communication with a remote computer;

maintaining synchronization of at least a second computer with said first computer, within said communication routines, wherein said second computer executes the same said computations as said first computer; and

migrating the connection endpoint, within said communication routines, from said first computer to said second computer in response to the failure of said first computer, providing transparent connection failover so that said remote computer still addresses said first computer but communicates with said second computer, and said remote computer is unaware of said server failure or of said connection failover.

29. A method as recited in claim 28, wherein said first computer and said second computer are configured for executing the same server application for performing said computations.

30. A method as recited in claim 29, wherein said server application is replicated on both said first and said second computers using active or semi-active replication.

31. A method as recited in claim 30, wherein said first computer and said second computer comprise a primary server and a backup server.

32. A method as recited in claim 31, wherein said remote computer comprises a client of said server.

33. A method as recited in claim 32, wherein said communication routines are configured to maintain communication between said client and one of said servers in response to at least one of said servers remaining operational.

34. A method as recited in claim 31, wherein said communication routines for said primary server and said backup are configured so that said server application undergoes identical state transitions eliminating the need to transfer application state in order to achieve transparent connection failover.

35. A method as recited in claim 31:

wherein said communication routines are configured for bringing up a new backup server, or returning a failed and repaired server to the system by executing a state transfer to said new backup server; and

wherein the application state and the communication infrastructure state are transferred in said state transfer.

36. A method as recited in claim 28, wherein said communication routines comprise routines for executing a connection-oriented communication protocol over a network communication protocol.

37. A method as recited in claim 36, wherein said connection-oriented protocol comprises the Transmission Control Protocol (TCP) and said network communication protocol comprises the Internet Protocol (IP).

38. A method as recited in claim 36, wherein said migrating connection endpoints is performed within bridge routines, within said communication routines, that execute in a bridge sublayer between said connection-oriented communication protocol layer and said network communication protocol layer.

39. A method as recited in claim 38:

wherein said bridge routines comprise primary server bridge routines and backup server bridge routines;

wherein said first computer and said second computer comprise a primary server and a backup server; and

wherein said remote computer comprises a client of said servers.

40. A method as recited in claim 39, further comprising maintaining a primary server output queue and a backup server output queue within said primary server bridge routines.

41. A method as recited in claim 40, wherein said primary server output queue and said backup server output queue are maintained for each fault-tolerant connection between said client and said servers.

42. A method as recited in claim 40, further comprising maintaining payload bytes in said primary server output queue within said primary server bridge, generated by said primary server'"'"'s connection-oriented communication protocol layer.

43. A method as recited in claim 40, further comprising maintaining payload bytes in said backup server output queue within said primary server bridge, where said bytes were communicated by said backup server to said primary server.

44. A method as recited in claim 43, wherein said communication routines for said primary server are configured to postpone acknowledging a client'"'"'s connection-oriented protocol segment until said primary server has received an acknowledgment of that segment from said backup server.

45. A method as recited in claim 44, wherein said communication routines for said backup server are configured for maintaining a copy of all connection-oriented protocol segments sent by said primary server to said client that said client has not acknowledged.

46. A method as recited in claim 40, wherein said communication routines for said primary server and said backup server are configured for removing said connection-oriented segment from said queues in response to receiving an acknowledgment of receipt of a connection-oriented segment by said client.

47. A method as recited in claim 39, wherein said bridge routines are configured to synchronize the connection-oriented protocol sequence numbers utilized by said primary server and said backup server.

48. A method as recited in claim 39, wherein said communication routines are configured to modify the connection endpoint address, from said primary server to said backup server, in response to the failure of said primary server, so that said client need not establish a new connection with said backup server.

49. A method as recited in claim 39, wherein said communication routines for said backup server are configured to stay within the Maximum Segment Size (MSS) and the maximum window size that were previously negotiated between said primary server and said client when establishing the connection.

50. A method as recited in claim 28:

wherein said communication routines are configured for executing a connection-oriented communication protocol over a network communication protocol;

wherein said connection-oriented protocol is the Transmission Control Protocol (TCP);

wherein said network communication protocol is the Internet Protocol (IP);

wherein communication bridge routines are inserted between the TCP layer and the IP layer of the TCP/IP stack; and

wherein said migration of connection endpoint is performed by said communication bridge routines.

51. A method as recited in claim 50, wherein said communication bridge routines are configured to generate between k+1 and 2k messages for a request, reply, or acknowledgment, in which k represents the number of server replicas.

52. A method as recited in claim 50, wherein said communication bridge routines are configured for performing said synchronization and said connection endpoint migration without proxies or communication routines in user space at each server.

53. A method as recited in claim 50, wherein said communication bridge routines are configured to perform said transparent connection failover without modifying network infrastructure, the server application, the client application, or the Transmission Control Protocol (TCP) layer or Internet Protocol (IP) layer of the TCP/IP stack of said client.

54. A method as recited in claim 50, wherein said communication bridge routines are configured for providing said transparent connection failover without replaying the entire history of the connection between said client and said primary server.

55. A method as recited in claim 50, wherein said bridge communication routines are configured to perform said transparent connection failover without modifying the Transmission Control Protocol (TCP) layer or the Internet Protocol (IP) layer of said server and without modifying the software or hardware of said client.

56. A method as recited in claim 50, wherein said communication bridge routines are configured to perform said transparent connection failover without requiring said backup server to change its Internet Protocol (IP) address to the address of the primary server and to send a gratuitous Address Resolution Protocol (ARP) request to announce that it can now be found at the Internet Protocol (IP) address of said primary server.

57. A method as recited in claim 28, wherein said communication bridge routines are configured to perform said transparent connection failover without requiring said client to establish a network connection with a dispatcher configured for routing all traffic from said client.

58. A method as recited in claim 28, wherein said communication bridge routines are configured to perform said transparent connection failover without utilizing a front-end layer-4 switch configured to route the packets to the appropriate server.

59. A method as recited in claim 28, wherein said communication bridge routines are configured to perform said transparent connection failover without preventing multicasting or forwarding of messages from said client.

60. A method as recited in claim 28, wherein said communication bridge routines are configured to perform said transparent connection failover without the need for primary and backup servers to be operational and cooperating during the transparent connection failover after the failure of the primary server.

61. A method as recited in claim 28, wherein said communication bridge routines are configured to perform said transparent connection failover without forcing all traffic to pass through a traffic redirector within a network router.

62. A method as recited in claim 28, wherein said communication bridge routines are configured to perform said transparent connection failover without performing context switches or protocol stack traversals in user space.

63. A method of ensuring transparent connection failover within a network, comprising:

executing communication routines on computers connected in said network so that computations can be executed within a server application on a primary server in response to communication with a client;

maintaining synchronization with at least one backup server executing said server application, within said communication routines, wherein said backup server executes the same computations as said primary server application; and

migrating the connection endpoint, within said communication routines, from said primary server to said backup server in response to the failure of said primary server, to provide transparent connection failover so that said client still addresses said primary server while actually communicating with one of said backup servers, and said client is unaware of said failure and of said failover;

wherein said server application is replicated using active or semi-active replication;

wherein said primary server and said backup server belong to the same subnet of said network;

wherein said primary server and said backup server applications undergo identical state transitions, eliminating the need for transfer of application state in order to achieve transparent connection failover; and

wherein said communication routines are configured to maintain communication between said client and one of said servers, provided that at least one of said servers remains operational.

64. A method of providing fault tolerance for two or more computers that communicate with a remote computer over a network connection and of providing transparent connection failover, comprising:

executing the same computations of same server application on said two or more computers in response to communication from a remote computer;

using the Transmission Control Protocol (TCP) over the Internet Protocol (IP); and

migrating the connection endpoint of one said computer upon the failure of that computer wherein the other of said computers continues to communicate with said remote computer.

65. A method as recited in claim 64, wherein said method requires no modifications to applications executing on said remote computers, or to the Transmission Control Protocol/Internet Protocol (TCP/IP) protocol stacks of said remote computers, or to other software or hardware of said remote computer, to achieve said transparent connection failover.

66. A method as recited in claim 64, wherein said communication between said remote computer and said two or more computers utilizes Transmission Control Protocol/Internet Protocol (TCP/IP) communication without the need for communication switches or routers dedicated to providing said transparent connection failover.

67. A method as recited in claim 64, wherein said transparent connection failover comprises detecting faults in said two or more computers, and the recovery from said faults to allow continued communication with said remote computers without any fault corrective actions being taken by said remote computer.

68. A method as recited in claim 67:

wherein said recovery from faults using said transparent connection failover is sufficiently rapid;

wherein sufficiently rapid recovery is achieved when the operation of said Transmission Control Protocol (TCP), said Internet Protocol (IP), and said remote computer, is not disrupted by said faults; and

a round-trip request/reply or message/acknowledgment, requiring two messages in standard Transmission Control Protocol/Internet Protocol (TCP/IP), requires from k+1 messages to at most 2k messages, where k is the number of said two or more computers that perform said same computations.

69. A method as recited in claim 64, wherein said two or more computers comprise clients that communicate with said remote computers acting as servers.

70. A method as recited in claim 64, wherein said two or more computers are configured to operate as clients when communicating with certain remote computers and servers when communicating with other remote computers.

71. A method as recited in claim 64, wherein said two or more computers are configured to send messages to said remote computers without a client/server relationship between them.

72. A method as recited in claim 64, wherein routines for maintaining said connection are performed within a bridge layer inserted into the Transmission Control Protocol/Internet Protocol (TCP/IP) stack, between the Transmission Control Protocol (TCP) layer and the Internet Protocol (IP) layer, without modification to either said TCP layer or said IP layer.

73. A method as recited in claim 64, wherein synchronization of said Transmission Control Protocol (TCP) layers, and of said Internet Protocol (IP) layers, is maintained between said two or more computers, and also between said two or more computers and said remote computer, so that operation of said TCP layer and said IP layer is not disrupted, if a fault in one of said two or more computers occurs, where said fault in one of said two or more computers requires operation of TCP and IP to be undertaken by another of said two or more computers.

74. A method as recited in claim 64, wherein said two or more computers receive all messages transmitted to said two or more computers by at least one of said remote computers, either directly, by using a promiscuous reception mode, or as retransmissions by one of said two or more computers that receive said messages directly.

75. A method as recited in claim 64, wherein one of said two or more computers is configured to operate as a primary computer and the others of said two or more computers are configured to operate as backup computers.

76. A method as recited in claim 75, wherein said bridge routines for maintaining a connection within any of said backup computers are configured to transmit data to said primary computer instead of said remote computer.

77. A method as recited in claim 76, wherein any of said backup computers transmits to said primary computer said data in addition to the address of said remote computer to which said Transmission Control Protocol (TCP) layer of said backup computer seeks to transmit said data.

78. A method as recited in claim 77:

wherein said Transmission Control Protocol (TCP) layer of said primary computer attaches a sequence number to said data when said Transmission Control Protocol (TCP) layer of said primary computer seeks to communicate data to said remote computer;

wherein said Transmission Control Protocol (TCP) layer of said backup computer attaches a sequence number to said data when said TCP layer of said backup computer seeks to communicate data to said remote computer;

wherein said sequence number attached to said data by said Transmission Control Protocol (TCP) layer of said primary computer might not be the same as said sequence number attached to corresponding data by said TCP layer of said backup computer.

79. A method as recited in claim 78:

wherein said primary computer maintains two buffers for each fault-tolerant connection with each of said remote computers;

wherein a first buffer is configured for containing data bytes that said Transmission Control Protocol (TCP) layer of said primary computer seeks to transmit, together with a sequence number for each data byte; and

wherein a second buffer is configured for containing data bytes that said Transmission Control Protocol (TCP) layer of said backup computer seeks to transmit, together with a sequence number for each data byte.

80. A method as recited in claim 79, wherein said routines for maintaining said fault-tolerant connection are executed in a bridge layer of said primary computer that adjusts said sequence numbers so that, when said bridge layer of said primary computer compares data in said first buffer with data in said second buffer, said bridge layer compares data that have the same adjusted sequence numbers and that are intended to be the same data.

81. A method as recited in claim 80, wherein said bridge layer of said primary computer is configured for comparing said sequence numbers of said data bytes in said first buffer and said sequence numbers of said data bytes in said second buffer, and in which, if said bridge layer finds different data with the same sequence number in said buffers, then said bridge layer reports an error to a fault handler.

82. A method as recited in claim 81, wherein said bridge layer of said primary computer, in response to finding the same data bytes with the same sequence numbers in said first buffer and said second buffer, transmits said data bytes to said remote computer.

83. A method as recited in claim 78:

wherein said primary computer maintains two buffers for each fault-tolerant connection with each of said remote computers;

wherein a first buffer is configured for containing data bytes that said TCP layer of said primary computer seeks to transmit to said remote computer, together with a sequence number for each data byte; and

wherein a second buffer is configured for containing the sequence numbers of said data bytes that said TCP layer of said backup computer seeks to transmit.

84. A method as recited in claim 83, wherein said bridge layer of said primary computer in response to finding the same sequence numbers in said first buffer and in said second buffer, transmits said data bytes, associated with said sequence numbers in said first buffer, to said remote computer.

85. A method as recited in claim 78:

wherein said Transmission Control Protocol (TCP) layer of said primary computer is configured to communicate data or an acknowledgment to said remote computer;

wherein said Transmission Control Protocol (TCP) layer of said primary computer determines an acknowledgment sequence number such that said primary computer has received all data bytes sent by said remote computer on said fault-tolerant connection with sequence numbers less than or equal to said acknowledgment sequence number;

wherein said Transmission Control Protocol (TCP) layer of said backup computer determines an acknowledgment sequence number, when said TCP layer of said backup computer seeks to communicate data to said remote computer, such that said backup computer has received all data bytes sent by said remote computer on said fault-tolerant connection with sequence numbers less than or equal to said acknowledgment sequence number; and

wherein said bridge layer of said primary computer transmits to said remote computer the smaller of said acknowledgment sequence numbers determined by said Transmission Control Protocol (TCP) layer of said primary computer and said TCP layer of said backup computer.

86. A method as recited in claim 85, wherein when said remote computer transmits an acknowledgment sequence number such that said remote computer has received all data bytes sent by said primary computer on said fault-tolerant connection with sequence numbers less than or equal to said acknowledgment sequence number, said acknowledgment sequence number is adjusted by said bridge layer of said primary computer before said acknowledgment sequence number is passed to said Transmission Control Protocol (TCP) layer of said primary computer, so that said TCP layer of said primary computer discards from its buffers all data bytes that said remote computer is acknowledging.

87. A method as recited in claim 86, wherein when said remote computer transmits an acknowledgment sequence number such that said remote computer has received all data bytes sent by said primary computer on said fault-tolerant connection with sequence numbers less than or equal to said acknowledgment sequence number, said acknowledgment sequence number is adjusted by said bridge layer of said backup computer, before said acknowledgment sequence number is passed to said Transmission Control Protocol (TCP) layer of said backup computer, so that said TCP layer of said backup computer discards from its buffers all data bytes that said remote computer is acknowledging.

88. A method as recited in claim 78:

wherein when said Transmission Control Protocol (TCP) layer of said primary computer seeks to communicate data or an acknowledgment to said remote computer, said TCP layer of said primary computer determines a window size such that said primary computer is willing to accept all TCP segments from said remote computer on said fault-tolerant connection with sequence numbers within that window;

wherein when said Transmission Control Protocol (TCP) layer of one of said backup computers seeks to communicate data or an acknowledgment to said remote computer, said TCP layer of one of said backup computers determines a window size such that said one of said backup computers is willing to accept all TCP segments from said remote computer on said fault-tolerant connection with sequence numbers within that window; and

wherein said bridge layer of said primary computer is configured for transmitting to said remote computer the smaller of said window sizes determined by said Transmission Control Protocol (TCP) layer of said primary computer and said TCP layers of said one or more backup computers.

89. A method as recited in claim 79, wherein when said primary computer determines that said backup computer has failed, said primary computer transmits to said remote computer all data present in said first buffer, and subsequently transmits to said remote computer all data subsequently passed by said Transmission Control Protocol (TCP) layer of said primary computer to said bridge layer of said primary computer, and that are addressed to said remote computer on said fault-tolerant connection.

90. A method as recited in claim 78, wherein when one of said backup computers determines that said primary computer has failed, one of said backup computers transmits to said remote computer all data transmitted by said backup computer to said bridge sublayer of said failed primary computer, and that are addressed to said remote computer for said fault-tolerant connection and that were not acknowledged by said remote computer, and subsequently transmits to said remote computer all data that are subsequently passed by said Transmission Control Protocol (TCP) layer of said backup computer to said bridge layer of said backup computer, and that are addressed to said remote computer on said fault-tolerant connection.

91. A method as recited in claim 90, wherein when a new computer, or a failed and repaired computer, starts to participate in said Transmission Control Protocol (TCP) connection as said backup computer, said primary computer checkpoints the state of its applications and also checkpoints the state of its TCP layer, as a single atomic action, and communicates said checkpoints to said new or repaired backup computer, and in which said new or repaired backup computer installs said application state and said TCP layer state as a single atomic action.

92. A method as recited in claim 91, wherein said primary computer, having communicated said checkpoints to said new or repaired backup computer, does not transmit to any remote computer any data received from said Transmission Control Protocol (TCP) layer of said primary computer until matching data have been received from said new or repaired backup computer.

93. A method as recited in claim 91, wherein said primary computer, when communicating said checkpoints to said new or repaired backup computer, also communicates the sequence number offset used by said bridge in said primary computer to adjust said sequence numbers for data bytes received from said TCP layer in said primary computer.

94. A method as recited in claim 93, wherein said backup computer, on receiving said sequence number offset from said primary computer, changes said sequence numbers in said TCP layer of said backup computer, so that said data bytes passed to said bridge by said TCP layer of said backup computer have exactly the same sequence numbers as said data bytes passed to said bridge by said TCP layer of said primary computer after said sequence numbers of said data bytes have been adjusted by said primary computer bridge.

95. A method as recited in claim 93, wherein said backup computer, on receiving said sequence number offset from said primary computer, is configured to cause said bridge in said backup computer to use said offset to adjust said sequence numbers attached to said data bytes received from said TCP layer of said backup computer so that said data bytes have exactly the same sequence numbers as said data passed to said bridge by said TCP layer of said primary computer after said sequence numbers of said data have been adjusted by said bridge of said primary computer, and also to adjust said acknowledgment sequence numbers received from said remote computers, so that said TCP layer of said backup computer discards from its buffers all data that said remote computer acknowledges.

96. A method as recited in claim 93:

wherein when the state of said Transmission Control Protocol (TCP) layer has been captured by said primary computer, the portions of said captured state corresponding to said sequence numbers are modified, so that said data bytes passed to said bridge by said TCP layer of said backup computer have exactly the same sequence numbers as said data passed to said bridge by said TCP layer of said primary computer after said sequence numbers of said data bytes have been adjusted by said bridge of said primary computer; and

wherein said modification can be performed within said primary computer or, alternatively, within said backup computer.