Method and system for a modular transmission control protocol (TCP) rare-handoff design in a streams based transmission control protocol/internet protocol (TCP/IP) implementation

• US 7,831,731 B2
• Filed: 06/12/2001
• Issued: 11/09/2010
• Est. Priority Date: 06/12/2001
• Status: Active Grant

• Current Assignee: Valtrus Innovations Limited (f/k/a Dolya Holdco 9 Limited) (Key Patent Innovations Limited)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Russell, Lance Warren, Tang, Wenting, Cherkasova, Ludmila
• Primary Examiner(s): Etienne; Ario
• Assistant Examiner(s): Todd; Gregory G
• Application Number:
  US09/880,631
• Publication Number:
  US 20020188740A1
• Time in Patent Office: 3,437 Days
• Field of Search: 709/201, 709/227, 709/223, 709/232, 709/235, 709/238, 709/244, 709/237, 709/222, 709/228
• US Class Current: 709/237
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1014 based on the content of a request
• CPC Class Codes: H04L 67/1017 based on a round robin mechanism
• CPC Class Codes: H04L 69/16 Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
• CPC Class Codes: H04L 69/161 Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields
• CPC Class Codes: H04L 69/163 In-band adaptation of TCP data exchange; In-band control procedures
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

A method and system for handing-off TCP states in a communication network. Specifically, the present invention allows for handing-off TCP states between nodes in an associated network that is optimized for rare handoff of TCP states. The handoff occurs between dynamically loadable modules that wrap around the TCP/IP stack located at a front-end node and a selected back-end web server. A handoff protocol implemented by the loadable modules works within the kernel level of the existing TCP/IP code. As such, no changes to the existing TCP/IP code is necessary. The loadable modules at the front-end are able to select a back-end web server depending on the content of the web request, coordinate handing off TCP states, and forward packets to the back-end web server. Loadable modules at the selected back-end modify response packets going out to reflect the proper TCP state of the front-end node.

Claims

1. In a communication network, a method of TCP state migration comprising the steps of:

a) establishing a TCP/IP communication session between a client computer and a first server computer, said first server computer part of a plurality of server computers forming a web cluster containing information, said communication session established for the transfer of data contained within said information;

b) handing off, by sending a handoff request packet, said communication session to a selected server computer from said first server computer over a persistent control channel using TCP handoff modules that are dynamically loadable within TCP/IP stacks in operating systems located at both said first server computer and said selected server computer, wherein the TCP handoff modules implement a TCP handoff protocol that works within kernel levels of the operating systems, wherein said first server computer includes a first bottom TCP (BTCP) module, and said selected server computer includes a second BTCP module; and

c) migrating a first TCP state of said first server computer to said selected server computer, and a second TCP state of said selected server computer to said first server computer over said control channel, wherein said migrating comprises migrating said first TCP state to said second BTCP module at said selected server computer over said control channel by including a first initial sequence number in said handoff request packet, such that said second BTCP module can calculate said first TCP state for said first server computer in said TCP/IP communication session.

2. The method as described in claim 1, wherein said step a) comprises the steps of:

receiving a SYN packet from said client computer at said first BTCP module located at said first server computer;

sending said SYN packet upstream to a first TCP module located above said first BTCP module in a first operating system of said first server computer;

receiving a first SYN/ACK packet from said first TCP module;

parsing said first TCP state from said first SYN/ACK packet, including said first initial sequence number for said first TCP module associated with said TCP/IP communication session;

sending said SYN/ACK packet to said client computer;

receiving an ACK packet from said client at said first BTCP module;

sending said ACK packet to said first TCP module;

receiving a web request packet associated with said TCP/IP communication session at said first BTCP module at said first server computer;

storing said SYN packet, said ACK packet and said web request packet at said first server computer.

3. The method as described in claim 2, wherein said step b) comprises the steps of:

examining content of said web request packet;

determining which of said plurality of server computers can best process said web request packet, based on said content, wherein the server computer determined to be able to best process said web request packet is the selected server computer;

sending the handoff request packet from said first BTCP module to said second BTCP module at said selected server computer over said control channel, if said selected server computer is not said first server computer;

including said SYN packet and said ACK packet in said handoff request packet;

changing a destination IP address of said SYN packet to a second IP address of said selected server computer, at said second BTCP module;

sending said SYN packet from said second BTCP module to a second TCP module at said selected server computer;

receiving a second SYN/ACK packet at said second BTCP module;

parsing said second TCP state from said second SYN/ACK packet, including a second initial sequence number, for said second TCP module, that is associated with said TCP/IP communication session;

changing a destination IP address of said ACK packet to said second IP address, at said second BTCP module;

updating said ACK packet to reflect said second TCP state of said selected server computer in said communication session;

sending said ACK packet that is updated from said second BTCP module to said second TCP module; and

sending a handoff acknowledgment message from said second BTCP module to said first BTCP module.

4. The method as described in claim 2, comprising the further steps of:

intercepting a connection indication message sent from said first TCP module to an application layer above said first TCP module at a first upper-TCP (UTCP) module at the first server computer, said connection indication message sent by said first TCP module upon establishing said communication session; and

holding said connection indication message at said first UTCP module, wherein said first UTCP module and said first BTCP module provide a wrapper around said first TCP module.

5. The method as described in claim 4, wherein said method comprises the further steps of:

sending a reset packet from said first BTCP module upon receiving said handoff acknowledgment message to said first TCP module;

discarding said connection indication message at said first UTCP module;

receiving incoming data packets from said client at said first BTCP module;

changing destination addresses of said incoming data packets to said second IP address;

updating sequence numbers and TCP checksum in said data packets to reflect said second TCP state of said selected server computer; and

forwarding said updated data packets to said selected server computer.

6. The method as described in claim 4, comprising the further steps of:

sending notification from said first BTCP module to said first UTCP module to release said connection indication message, if said selected server computer is said first server computer;

sending incoming data packets, including said web request packet, from said client computer, received at said first BTCP module, upstream.

7. The method as described in claim 4, wherein the first BTCP module and first UTCP module are dynamically loadable kernel modules.

8. The method as described in claim 2, comprising the further steps of:

intercepting outgoing response packets from said selected server computer at a second bottom TCP (BTCP) module located at said selected server computer;

changing, by the second BTCP module, source addresses of said response packets to a first IP address of said first server computer;

updating, by the second BTCP module, sequence numbers and TCP checksum in said response packets to reflect said first TCP state of said first server computer; and

sending said updated response packets to said client computer without passing the updated response packets through the first server computer.

9. The method as described in claim 8, comprising the further steps of:

monitoring TCP/IP control traffic for said communication session at said second BTCP module;

understanding when said communication session is closed at said selected server computer;

sending a termination message to said first server computer over said control channel;

terminating said TCP/IP communication session at said first server computer by terminating a forwarding mode at said first BTCP module; and

freeing data resources associated with said communication session at said first server computer.

10. The method as described in claim 1, wherein step c) comprises:

migrating a second TCP state of said selected server computer to said first BTCP module by including said second initial sequence number in a handoff acknowledgment message, such that said first BTCP module can calculate said second TCP state for said selected server computer in said TCP/IP communication session.

11. The method as described in claim 1, wherein the TCP handoff modules are dynamically loadable kernel modules, the method further comprising:

dynamically loading the TCP handoff modules in the corresponding first server computer and selected server computer without modifying the operating systems of the respective first server computer and selected server computer.

12. In a communication network, a method of TCP state migration comprising the steps of:

a) establishing a TCP/IP communication session between a client computer and a first server computer, said first server computer part of a plurality of server computers forming a web cluster containing information, said communication session established for the transfer of data contained within said information;

b) monitoring traffic associated with establishing said TCP/IP communication session to understand a first initial TCP state of said first server computer associated with said TCP/IP communication session, at a first bottom-TCP (BTCP) module at said first server computer, wherein said first BTCP module is a dynamically loadable kernel module (DLKM) loaded in said first server computer without modifying a first operating system of said first server computer;

c) receiving a web request associated with said TCP/IP communication session at said first BTCP module at said first server computer;

d) examining content of said web request;

e) determining which of said plurality of server computers, a selected server computer, can best process said web request, based on said content;

f) handing off said communication session to said selected server computer from said first server computer over a persistent control channel, if said selected server computer is not said first server computer;

g) monitoring traffic associated with handing off said TCP/IP communication session to understand a second initial TCP state of said selected server computer associated with said TCP/IP communication session, at a second BTCP module at said selected server computer, wherein said second BTCP module is a DLKM loaded in said selected server computer without modifying a second operating system of said selected server computer;

h) migrating said first initial TCP state to said selected server computer over said control channel, such that said second BTCP module can calculate a first TCP state for said first server computer in said TCP/IP communication session;

i) sending a second initial TCP state of said selected server computer to said first BTCP module, such that said first BTCP module can calculate a second TCP state for said selected server computer in said TCP/IP communication session;

j) forwarding data packets received at said first BTCP module from said client to said selected server computer, by changing said data packets to reflect said second TCP state and a second IP address of said selected server computer;

k) sending response packets from said selected server computer directly to said client computer by changing said response packets to reflect said first TCP state and a first IP address of said first server computer; and

l) terminating said TCP/IP communication session at said first server computer when said TCP/IP communication session is closed.

13. The method as described in claim 12, wherein said steps a) and b) comprise the steps of:

receiving a SYN packet from said client computer at said first BTCP module;

sending said SYN packet upstream to a first TCP module located above said first BTCP module in said first operating system of said first server computer;

receiving a first SYN/ACK packet from said first TCP module;

parsing, by the first BTCP module, said first initial TCP state from said first SYN/ACK packet, including a first initial sequence number for said first TCP module associated with said TCP/IP communication session;

sending said SYN/ACK packet to said client computer;

receiving, by the first BTCP module, an ACK packet from said client computer at said first BTCP module;

sending said ACK packet to said first TCP module;

storing said SYN packet, said ACK packet and said web request at said first server computer.

14. The method as described in claim 13, wherein said steps f) and g) comprise the steps of:

sending a handoff request packet from said first BTCP module to said second BTCP module over said control channel;

including said SYN packet and said ACK packet in said handoff request packet;

changing a destination IP address of said SYN packet to a second IP address of said selected server computer, at said second BTCP module;

sending, by said second BTCP module, said SYN packet to a second TCP module at said selected server computer;

receiving a second SYN/ACK packet at said second BTCP module;

parsing said second initial TCP state from said second SYN/ACK packet, including a second initial sequence number, for said second TCP module, that is associated with said TCP/IP communication session;

changing a destination IP address of said ACK packet to said second IP address, at said second BTCP module;

updating said ACK packet to reflect said second TCP state of said selected server computer in said communication session;

sending said ACK packet that is updated from said second BTCP module to said second TCP module; and

sending a handoff acknowledgment message from said second BTCP module to said first BTCP module.

15. The method as described in claim 14, wherein said ACK packet includes said first initial TCP state of said first server computer.

16. The method as described in claim 14, wherein said handoff acknowledgment message includes said second initial sequence number.

17. The method as described in claim 14, comprising the further steps of:

intercepting a connection indication message sent from said first TCP module to an application layer above said first TCP module at a first upper-TCP (UTCP) module at said first server computer, said connection indication message sent by said first TCP module upon establishing said communication session; and

holding said connection indication message at said first UTCP module, wherein said first UTCP module and said first BTCP module provide a wrapper around said first TCP module.

18. The method as described in claim 17, comprising the further steps of:

sending a reset packet from said first BTCP module upon receiving said handoff acknowledgment message to said first TCP module;

discarding said connection indication message at said first UTCP module;

receiving incoming data packets from said client at said first BTCP module;

changing destination addresses of said incoming data packets to said second IP address;

updating sequence numbers and TCP checksum in said data packets to reflect said second TCP state of said selected server computer; and

forwarding said updated data packets to said selected server computer.

19. The method as described in claim 17, comprising the further steps of:

sending notification from said first BTCP module to said first UTCP module to release said connection indication message, if said selected server computer is said first server computer; and

sending incoming data packets, including said web request, from said client computer, received at said first BTCP module, upstream.

20. The method as described in claim 12, wherein step k) comprises the steps of:

intercepting outgoing response packets from said selected server computer at said second BTCP module;

changing, by said second BTCP module, source addresses of said response packets to said first IP address;

updating, by said second BTCP module, sequence numbers and TCP checksum in said response packets to reflect said first TCP state of said first server computer; and

sending said updated response packets to said client computer without passing the updated response packets through said first server computer.

21. The method as described in claim 12, wherein step l) comprises the steps of:

monitoring TCP/IP control traffic for said communication session at said second BTCP module;

understanding when said communication session is closed at said selected server computer;

sending a termination message to said first server computer over said control channel;

terminating a forwarding mode at said first BTCP module; and

freeing data resources associated with said communication session at said first server computer.

22. The method as described in claim 12, wherein each of said plurality of server computers is constructed similarly including BTCP modules located downstream from respective TCP modules in the corresponding server computers, and UTCP modules located upstream from the corresponding TCP modules, wherein each pair of the BTCP modules and UTCP modules provides a wrapper around a corresponding TCP module.

23. The method as described in claim 22, wherein said control channel allows for communication between all UTCP modules of the corresponding server computers.

24. The method as described in claim 12, wherein said plurality of server computers is coupled together over a wide area network in said communication network.

25. The method as described in claim 12, wherein said information is partitioned/partially replicated throughout each of said plurality of server computers.

26. A first server computer comprising:

at least one processor;

an operating system executable on the at least one processor;

a TCP module in the operating system;

a first upper TCP (UTCP) module located above the TCP module in the operating system of said first server computer;

a first bottom TCP (BTCP) module located below said TCP module, wherein said first UTCP, TCP, and first BTCP modules implement handing off a communication session between the first server computer and a second node, wherein the first BTCP and first UTCP modules are dynamically loadable kernel modules (DLKMs) that are dynamically loadable and unloadable in said first server computer without modifying the operating system,

wherein said communication session is established between a client computer and said first server computer, said first server computer part of a plurality of server computers including said second node forming a cluster that contains information, said communication session established for the transfer of data contained within said information;

wherein said first BTCP module is configured to;

receive a web request associated with said TCP/IP communication session;

examine content of said web request;

determine which of said plurality of server computers can best process said web request, based on said content, wherein the server computer determined to be able to best process said web request is the second node;

hand off said communication session to said second node from said first server computer over a persistent control channel, if said second node is not said first server computer, wherein the handing off causes migration of a first TCP state of said first server computer to the second node, and migration of a second TCP state of said second node to the server computer.

27. The first server computer as described in claim 26, wherein said first BTCP module is configured to further:

receive a SYN packet from said client computer;

send said SYN packet upstream to said TCP module;

receive a first SYN/ACK packet from said TCP module;

parse a first initial TCP state from said first SYN/ACK packet, including a first initial sequence number for said TCP module associated with said TCP/IP communication session;

send said SYN/ACK packet to said client computer;

receive an ACK packet from said client computer;

send said ACK packet to said TCP module.

28. The first server computer as described in claim 27, wherein said first BTCP module is configured to further:

send a handoff request packet to a second BTCP module over said control channel, said second BTCP module located below a second TCP module in a second operating system at said second node;

include said SYN packet and said ACK packet in said handoff request packet;

receive a handoff acknowledgment message from said second BTCP module.

29. The first server computer as described in claim 28, wherein said first UTCP module is configured to:

intercept a connection indication message sent from said TCP module to an application layer above said TCP module, said connection indication message sent by said TCP module upon establishing said communication session; and

hold said connection indication message.

30. The first server system as described in claim 29, wherein said first BTCP module is configured to further:

send a reset packet upon receiving said handoff acknowledgment message to said TCP module;

receive incoming data packets from said client computer;

change destination addresses of said incoming data packets to a second IP address of said second node;

update sequence numbers and TCP checksum in said data packets to reflect said second TCP state of said second node; and

forward said updated data packets to said second node.

31. The first server computer as described in claim 29, wherein said first BTCP module is configured to further:

send notification to said first UTCP module to release said connection indication message, if said second node is said first server computer;

send incoming data packets, including said web request, from said client computer, received at said first BTCP module, upstream.

32. The first server computer as described in claim 26, wherein said first BTCP module is configured to:

receive a handoff request from a second BTCP module located at a second server computer over a persistent control channel, said second server computer having established a second communication session with a second client computer, said handoff request including a SYN packet and an ACK packet, said SYN packet and ACK packet used for establishing said second communication session between said second client computer and said second server computer, said ACK packet including a first initial TCP state including a first initial TCP sequence number of said second server computer in said communication session;

change a destination IP address of said SYN packet to a first IP address of said first server computer;

send said SYN packet to said TCP module;

receive a SYN/ACK packet;

parse a second initial TCP state from said SYN/ACK packet, including a second initial sequence number, for said TCP module;

change a destination IP address of said ACK packet to said first IP address;

update said ACK packet to reflect said second TCP state;

send said ACK packet that is updated to said TCP module; and

send a handoff acknowledgment message to said second BTCP module over said control channel.