Method and apparatus for traversing a translation device with a security protocol

• US 7,346,770 B2
• Filed: 10/31/2002
• Issued: 03/18/2008
• Est. Priority Date: 10/31/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Swander, Brian D., Dixon, William H.
• Primary Examiner(s): Sheikh; Ayaz
• Assistant Examiner(s): DOAN, TRANG T
• Application Number:
  US10/284,931
• Publication Number:
  US 20040088537A1
• Time in Patent Office: 1,965 Days
• Field of Search: 713/171, 726 14- 15
• US Class Current: 713/153
• CPC Class Codes: H04L 63/0464 using hop-by-hop encryption, i.e. wherein an intermediate entity decrypts the information and re-encrypts it before forwarding it
• CPC Class Codes: H04L 63/164 at the network layer

Abstract

The invention uses a three phase IKE protocol main mode negotiation to implement a port float algorithm that permits UDP encapsulated ESP traffic to traverse an IPSec-aware NAT. The NAT is connected to a plurality of client computers on a private network and provides an interface between the client computers and a server connected to a public network. In a first phase, a client and the server determine whether both are capable of sending UDP encapsulated ESP packets. In a second phase, the client and server conduct NAT discovery and determine whether the client, server, or both operate behind a NAT. In a third phase, the client and server initiate a port float algorithm, moving a destination UDP port specified in IKE packets from a first port value to a second port value. The server maintains a data structure that allows the server to identify the client sending IKE packets after exiting the second phase and entering the third phase.

Claims

1. A method for sending user datagram protocol encapsulated encapsulating security protocol packets through a network address translation device on a private network from a client on the private network to a server on a public network, using a key management and exchange protocol negotiation, comprising:

determining whether both the client and server are capable of sending the user datagram protocol encapsulated encapsulating security protocol packets, wherein the client sends a first key management and exchange protocol packet to the server and receives a second key management and exchange protocol packet from the server ,the first and second key management and exchange protocol packet send over first source and destination user datagram protocol port;

creating an entry in a data structure that uniquely identifies a connection between the client and the server exchanging key management and exchange protocol packets sent over the first source and destination user datagram protocol ports, the entry including at least an internet protocol address of the client and an internet protocol address of the server;

determining whether at least one of the client or the server operate behind the network address translation device; and

if it is determined that at least one of the client or the server operate behind the network address translation device;

selecting second source and destination ports, the second source and destination ports being distinct from the first source and destination ports; and

sending the user datagram protocol encapsulated encapsulating security protocol packets over the second source and destination ports so that the user datagram protocol encapsulated encapsulating security protocol packets are able to traverse the network address translation device, wherein the server identifies the client using the data structure;

wherein the network address translation device interprets the user datagram protocol encapsulated encapsulating security protocol packets designating the first destination port as key management and exchange protocol packets and user datagram protocol encapsulated encapsulating security protocol packets designating the second destination port as non-key management and exchange protocol packets.

2. The method of claim 1, wherein the network address translation device interprets the user datagram protocol encapsulated encapsulating security protocol packets designating the first destination port as invalid and user datagram protocol encapsulated encapsulating security protocol packets designating the second destination port as valid.

3. The method of claim 1, wherein the first and second key management and exchange protocol packets each include a vendor ID;

the vendor ID being a known value indicating a capability of sending user datagram protocol encapsulated encapsulating security protocol packets.

4. The method of claim 1, wherein the step of determining whether at least one of the client or the server operate behind the network address translation device further comprises:

sending a third key management and exchange protocol packet including a first and second network address translation device discovery payload, the first network address translation device T discovery payload including a hash function of the first source port address and a source internet protocol address as known to the client and the second network address translation device discovery payload including a hash function of the first destination port address and a destination internet protocol address as known to the client;

receiving a fourth key management and exchange protocol packet including a third and a fourth network address translation device discovery payload, the third network address translation device discovery payload including a hash function of the first source port address and a source internet protocol address as known to the server and the fourth network address translation device discovery payload including a hash function of the first destination port address and a destination internet protocol address as known to the server; and

determining based on a comparison of the first and third network address translation device discovery payloads that the client operates behind the network address translation device.

5. The method of claim 4 further comprising:

determining based on a comparison of the second and fourth network address translation device discovery payloads that the server does not operate behind the network address translation device.

6. The method of claim 1, wherein the key management and exchange protocol negotiation is a main mode negotiation.

7. The method of claim 1 wherein the mapping data structure further comprises an I-Cookie value assigned by the client and an R-Cookie value assigned by the server.

8. The method of claim 1, further comprising sending a request for a new cryptographic key in third key management and exchange protocol packet using the second source and destination ports.

9. A method for receiving user datagram protocol encapsulated encapsulating security protocol packets at a server on a public network, the user datagram protocol encapsulated encapsulating security protocol packets being sent from a client operating behind a network address translation device on a private network, comprising:

receiving a first key management and exchange protocol packet and sending a second key management and exchange protocol packet, the first and second key management and exchange protocol packets designating a first destination port and including a vendor identification value indicating a capability to send the user datagram protocol encapsulated encapsulating security protocol packets;

storing a value identifying a connection between the client and the server exchanging key management and exchange protocol packets designating the first destination port;

determining that at least the client operates behind the network address translation device; and

if it is determined that at least the client operates behind the network address translation device;

selecting a second destination port, the second destination port being distinct from the first destination port;

receiving a third key management and exchange protocol packet designating the second destination port;

determining that the third key management and exchange protocol packet is sent by the client by comparing a unique identification within the third key management and exchange protocol packet to the stored value; and

receiving the user datagram protocol encapsulated encapsulating security protocol packets sent over the second destination port, the user datagram protocol encapsulated encapsulating security protocol packets designating the second destination port, so that the user datagram protocol encapsulated encapsulating security protocol packets are able to traverse the network address translation device.

10. The method of claim 9, wherein the step of determining that at least the client operates behind a network address translation device comprises:

receiving a discovery payload from the client, the discovery payload including a first value derived from a hash function of a client IP and port address as known to the client;

calculating a second value from a hash function of the client IP and port address as known to the server; and

comparing the first value to the second value and determining that the client operates behind the network address translation device.

11. The method of claim 10, wherein the first value is distinct from the second value.

12. The method of claim 9, wherein the stored valued comprises an I-Cookie value assigned by the client, an R-Cookie value assigned by the server, an internet protocol address of the client and an internet protocol address of the server.

13. Computer storage media having computer executable instructions for sending user datagram protocol encapsulated encapsulating security protocol packets through a network address translation device on a private network from a client on the private network to a server on a public network, using a key management and exchange protocol negotiation, comprising:

determining whether both the client and server are capable of sending the user datagram protocol encapsulated encapsulating security protocol packets, wherein the client sends a first key management and exchange protocol packet to the server and receives a second key management and exchange protocol packet from the server, the first and second key management and exchange protocol packets sent over first source and destination user datagram protocol ports;

creating an entry in a data structure that uniquely identifies a connection between the client and the server exchanging key management and exchange protocol packets sent over the first source and destination user datagram protocol ports, the entry including at least an internet protocol address of the client and an internet protocol address of the server;

determining whether at least one of the client or the server operate behind the network address translation device; and

if it is determined that at least one of the client or the server operate behind the network address translation device;

selecting second source and destination ports, the second source and destination ports being distinct from the first source and destination ports; and

sending the user datagram protocol encapsulated encapsulating security protocol packets over the second source and destination ports so that the user datagram protocol encapsulated encapsulating security protocol packets are able to traverse the network address translation device, wherein the server identifies the client using the data structure.

14. The computer storage media of claim 13, wherein the first and second key management and exchange protocol packets each include a vendor ID;

the vendor ID being a known value indicating a capability of sending user datagram protocol encapsulated encapsulating security protocol packets.

15. The computer storage media of claim 13, wherein determining whether at least one of the client or the server operate behind the network address translation device comprises;

sending a third key management and exchange protocol packet including a first and second network address translation device discovery payload, the first network address translation device discovery payload including a hash function of the first source port address and a source internet protocol address as known to the client and the second network address translation device discovery payload including a hash function of the first destination port address and a destination internet protocol address as known to the client;

receiving a fourth key management and exchange protocol packet including a third and a fourth network address translation device discovery payload, the third network address translation device discovery payload including a hash function of the first source port address and a source internet protocol address as known to the server and the fourth network address translation device discovery payload including a hash function of the first destination port address and a destination internet protocol address as known to the server; and

determining based on a comparison of the first and third network address translation device discovery payloads that the client operates behind the network address translation device.

16. The computer storage media of claim 13 wherein the data structure further comprises an I-Cookie value assigned by the client and an R-Cookie value assigned by the server.

17. The computer storage media of claim 13 further comprising sending a request for a new cryptographic key in a third key management and exchange protocol packet using the second source and destination port addresses.

18. Computer storage media having computer executable instructions for receiving user datagram protocol encapsulated encapsulating security protocol packets at a server on a public network, the user datagram protocol encapsulated encapsulating security protocol packets being sent from a client operating behind a network address translation device on a private network, comprising:

receiving a first key management and exchange protocol packet and sending a second key management and exchange protocol packet, the first and second key management and exchange protocol packets designating a first destination port and including a vendor identification value indicating a capability to send the user datagram protocol encapsulated encapsulating security protocol packets;

storing a value identifying a connection between the client and the server exchanging key management and exchange protocol packets designating the first destination port;

determining that at least the client operates behind the network address translation device; and

if it is determined that at least the client operates behind the network address translation device;

selecting a second destination port, the second destination port being distinct from the first destination port;

receiving a third key management and exchange protocol packet designating the second destination port;

determining that the third key management and exchange protocol packet is sent by the client by comparing a unique identification within the third key management and exchange protocol packet to the stored value; and

receiving user datagram protocol encapsulated encapsulating security protocol packets sent over the second destination port, the user datagram protocol encapsulated encapsulating security protocol packets designating the second destination port, so that the user datagram protocol encapsulated encapsulating security protocol packets are able to traverse the network address translation device.

19. The computer storage media of claim 18, wherein the step of determining that at least the client operates behind a network address translation device comprises:

receiving a discovery payload from the client, the discovery payload including a first value derived from a hash function of a client IP and port address as known to the client;

deriving a second value from a hash function of the client IP and port address as known to the server;

comparing the first value to the second value and determining that the client operates behind the network address translation device.

20. The computer storage media of claim 19, wherein the stored valued comprises an I-Cookie value assigned by the client, an R-Cookie value assigned by the server, an internet protocol address of the client and an internet protocol address of the server.