System and method for nesting virtual private networking connections with coincident endpoints

• US 6,978,308 B2
• Filed: 03/21/2001
• Issued: 12/20/2005
• Est. Priority Date: 03/21/2001
• Status: Expired due to Term

• Current Assignee: Trend Micro Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Boden, Edward B., Gruber, Franklin A.
• Primary Examiner(s): VU, THONG H
• Application Number:
  US09/813,911
• Publication Number:
  US 20020138623A1
• Time in Patent Office: 1,735 Days
• Field of Search: 709/202, 709/224, 709/227, 709/237, 709/225, 709/226, 709/229, 709/249, 709/200, 713/153, 713/200, 713/201, 370/389, 370/237, 370/256, 370/392, 707/10, 380/23, 340/5
• US Class Current: 709/229
• CPC Class Codes: H04L 12/4633 Interconnection of networks using encapsulation techniques, e.g. tunneling
• CPC Class Codes: H04L 63/0272 Virtual private networks
• CPC Class Codes: H04L 63/164 at the network layer
• CPC Class Codes: H04L 69/24 Negotiation of communication capabilities
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

A communication network includes a plurality of nodes, selectively including a client, a remote gateway Internet service provider, the Internet, a local enterprise gateway, and an enterprise internal network. A local coincident endpoint is established at a first node for an outer connection with a remote node and an inner connection with a different remote node. The nodes participate in negotiations on the outer connection to set up the inner connection as a secure connection. Thereafter, responsive to communications on the inner connection, the first node establishes links to the outer connection selectively to receive or send communications double nested on the outer connection.

Claims

1. Method for nesting IP Sec-based VPN connections between a plurality of nodes in a communication network in which nested connections establish a tunnel within a tunnel including an inner connection and an outer connection having at least one coincident endpoint residing on a same node, comprising the steps of:

receiving at a first node on said outer connection a request from a second node to establish a coincident endpoint for nesting a secure inner connection within said outer connection;

negotiating over said outer connection parameters defining said inner connection and resulting from Internet key exchange (IKE) negotiations for establishing an agreed upon encryption algorithm and key generation; and

thereafter

responsive to communication occurring on said inner connection, at said first node linking said inner connection to said outer connection for selectively receiving and sending said communication double nested on said outer connection to allow subsequent traffic to be correctly processed by said inner connection, then by said outer connection, at both ends of both connections and thereby enabling outbound traffic between respective nodes selectively to flow inside said outer tunnel and not said inner tunnel, in said inner tunnel and said outer tunnel, and in neither tunnel.

2. The method of claim 1, further comprising the step of using Layer 2 Tunnel Protocol (L2TP) to tunnel packets across said communication network.

3. Method for operating an enterprise gateway node to a plurality of nodes in a communication network in which nested connections establish an inner tunnel within an outer tunnel including an inner connection and an outer connection having at least one coincident endpoint residing on a said gateway node, comprising the steps of:

receiving at said gateway node from a remote client node a request to establish an outer connection;

receiving at said gateway over said outer connection a request to establish, and thereupon negotiating parameters establishing, a secure inner connection using Internet key exchange (IKE) negotiations for establishing an agreed upon encryption algorithm and key generation and further including establishing a local coincident endpoint of said inner and outer connections at said gateway;

responsive to outbound or inbound traffic on said inner connection, establishing links to said outer connection for communicating said traffic double nested on said outer connection to allow subsequent traffic to be correctly processed by said inner connection, then by said outer connection, at both ends of both connections and thereby enabling outbound traffic between respective nodes selectively to flow inside said outer tunnel and not said inner tunnel, in said inner tunnel and said outer tunnel, and in neither tunnel.

4. The method of claim 3, further comprising the step of:

tunneling packets across said communication network using Layer 2 Tunnel Protocol (L2TP).

5. A method for operating a first one of a plurality of nodes in a communications network in which nested connections establish an inner tunnel within an outer tunnel including an inner connection and an outer connection having at least one coincident endpoint residing on said first node, comprising the steps of:

establishing at said first node a coincident endpoint for an outer connection and an inner connection with at least one second node in said network for setting up a tunnel within a tunnel between said first and second nodes and executing Internet key exchange (IKE) negotiations for establishing an agreed upon encryption algorithm and key generation;

responsive to starting communication of traffic over said connections, establishing a link from said inner connection to said outer connection including establishing a local coincident endpoint of said inner and outer connections at said first node; and

responsive to said links, selectively encapsulating said traffic to said outer connection for transfer to said second node and decapsulating said traffic from said outer connection followed by decapsulating said traffic from said inner connection for receipt at said first node.

6. The method of claim 5, said inner connection being a secure connection.

7. The method of claim 5, further comprising the step of:

tunneling packets across said communication network using Layer 2 Tunnel protocol (L2TP).

8. Method for nesting connections in a tunnel within a tunnel having at least one coincident endpoint between a plurality of nodes in a communication network, said nodes including a client, an Internet service provider (ISP), an enterprise gateway, and an internal network, comprising the steps of:

operating said client node to call said ISP node;

operating said ISP node to start an outer connection with respect to said gateway node and to return an IP address to said client node;

operating said client node to send to said gateway node over said outer connection a request to establish a secure nested inner connection;

operating said client node and said gateway node to negotiate over said outer connection parameters defining said secure nested inner connection resulting from Internet key exchange (IKE) negotiations for establishing an agreed upon encryption algorithm and key generation, and saving said parameters at said gateway node; and

thereafter

operating said client node to start said inner connection;

operating said ISP node to decapsulate said outer connection;

operating said client node to decapsulate said inner connection; and

operating said gateway node to recognize the start of said inner connection and to link said inner connection to said outer connection to allow subsequent traffic to be correctly processed by said inner connection, then by said outer connection, at both ends of both connections, and sending outbound traffic in said inner connection double nested in said outer connection.

9. The method of claim 8, further comprising the step of:

tunneling packets across said communication network using Layer 2 Tunnel protocol (L2TP).

10. System for nesting connections between a plurality of nodes in a communication network in which nested connections establish a tunnel within a tunnel including an inner connection and an outer connection having at least one coincident endpoint residing on a same node, comprising:

a first node on an outer connection for receiving a request from a second node to establish a coincident endpoint for nesting an inner connection within said outer connection including executing Internet key exchange (IKE) negotiations for establishing an agreed upon encryption algorithm and key generation;

said first and second nodes negotiating over said outer connection parameters defining said inner connection; and

thereafter

said first node being responsive to communication occurring on said inner connection for linking to said outer connection for selectively receiving or sending said communication double nested on said outer connection to allow subsequent traffic to be correctly processed by said inner connection, then by said outer connection, at both ends of both connections;

thereby enabling outbound traffic between respective nodes selectively to flow inside said outer tunnel and not said inner tunnel, in said inner tunnel and said outer tunnel, and in neither tunnel.

11. The system of claim 10, said inner connection being a secure connection.

12. The system of claim 11, said inner connection being an IPsec connection.

13. The system of claim 12, further comprising a Layer 2 Tunnel protocol (L2TP) connection for tunneling packets across said communication network.

14. A program storage device readable by a machine, tangibly embodying a program of instructions executable by a machine to perform method steps for nesting connections between a plurality of nodes in a communication network in which nested connections establish a tunnel within a tunnel including an inner connection and an outer connection having at least one coincident endpoint residing on a same node, said method steps comprising:

receiving at a first node on an outer connection a request from a second node to establish a coincident endpoint for nesting an inner connection within said outer connection;

negotiating over said outer connection parameters defining said inner connection resulting from Internet key exchange (IKE) negotiations for establishing an agreed upon encryption algorithm and key generation; and

thereafter

responsive to communication occurring on said inner connection, at said first node linking to said outer connection for selectively receiving or sending said communication double nested on said outer connection to allow subsequent traffic to be correctly processed by said inner connection, then by said outer connection, at both ends of both connections.

15. A program storage device readable by a machine, tangibly embodying a program of instructions executable by a machine to perform method steps for operating an enterprise gateway in a communications network in which nested connections establish a tunnel within a tunnel including an inner connection and an outer connection having at least one coincident endpoint residing on a same node, said method steps comprising:

receiving at said gateway from a remote client a request to establish an outer connection;

receiving at said gateway over said outer connection a request to establish, and thereupon negotiating parameters including executing Internet key exchange (IKE) negotiations for establishing an agreed upon encryption algorithm and key generation for establishing, a secure inner connection;

responsive to outbound or inbound traffic on said inner connection, establishing links to said outer connection for communicating said traffic double nested on said outer connection to allow subsequent traffic to be correctly processed by said inner connection, then by said outer connection, at both ends of both connections thereby enabling outbound traffic between respective nodes selectively to flow inside said outer tunnel and not said inner tunnel, in said inner tunnel and said outer tunnel, and in neither tunnel.

16. A program storage device readable by a machine, tangibly embodying a program of instructions executable by a machine to perform method steps for operating a first one of a plurality of nodes in a communications network in which nested connections establish a tunnel within a tunnel including an inner connection and an outer connection having at least one coincident endpoint residing on a same node, comprising the steps of:

establishing at said first node a coincident endpoint for an outer connection and an inner connection with at least one second node in said network;

responsive to starting communication of traffic over said connections, establishing a link from said inner connection to said outer connection including executing Internet key exchange (IKE) negotiations for establishing an agreed upon encryption algorithm and key generation; and

responsive to said links, selectively encapsulating said traffic to said outer connection for transfer to said second node or decapsulating said traffic from said outer connection for receipt at said first node to allow subsequent traffic to be correctly processed by said inner connection, then by said outer connection, at both ends of both connections.

17. A computer program product for nesting connections between a plurality of nodes in a communication network in which nested connections establish a tunnel within a tunnel including an inner connection and an outer connection having at least one coincident endpoint residing on a same node, said computer program product comprising:

a digital recording medium;

first program instructions for receiving at a first node on an outer connection a request from a second node to establish a coincident endpoint for nesting an inner connection within said outer connection;

second program instructions for negotiating over said outer connection parameters defining said inner connection resulting from Internet key exchange (IKE) negotiations for establishing an agreed upon encryption algorithm and key generation; and

thereafter

third program instructions, responsive to communication occurring on said inner connection, at said first node linking to said outer connection for selectively receiving or sending said communication double nested on said outer connection to allow subsequent traffic to be correctly processed by said inner connection, then by said outer connection, at both ends of both connections;

thereby enabling outbound traffic between respective nodes selectively to flow inside said outer tunnel and not said inner tunnel, in said inner tunnel and said outer tunnel, and in neither tunnel; and

wherein

said first, second and third program instructions are recorded on said digital recording medium.

18. A computer program product for operating an enterprise gateway node to a network in which nested connections establish a tunnel within a tunnel including an inner connection and an outer connection having at least one coincident endpoint residing on said gateway node, said computer program product, comprising:

a digital recording medium;

first program instructions for receiving at said gateway from a remote client a request to establish an outer connection;

second program instructions for receiving at said gateway over said outer connection a request to establish, and thereupon negotiating parameters establishing, a secure inner connection resulting from Internet key exchange (IKE) negotiations for establishing an agreed upon encryption algorithm and key generation;

third program instructions, responsive to outbound or inbound traffic on said inner connection, for establishing links to said outer connection for communicating said traffic double nested on said outer connection to allow subsequent traffic to be correctly processed by said inner connection, then by said outer connection, at both ends of both connections; and

wherein

said first, second, and third program instructions are recorded on said digital recording medium.

19. A computer program product for operating a first one of a plurality of nodes in a communications network in which nested connections establish a tunnel within a tunnel including an inner connection and an outer connection having at least one coincident endpoint residing on a same node said computer program product comprising:

a magnetic recording medium;

first program instructions for establishing at said first node a coincident endpoint for an outer connection and an inner connection with at least one second node in said network;

second program instructions, responsive to starting communication of traffic over said connections, for executing Internet key exchange (IKE) negotiations for establishing an agreed upon encryption algorithm and key generation and establishing a link from said inner connection to said outer connection; and

third program instructions, responsive to said links, for selectively encapsulating said traffic to said outer connection for transfer to said second node or decapsulating said traffic from said outer connection for receipt at said first node to allow subsequent traffic to be correctly processed by said inner connection, then by said outer connection, at both ends of both connections; and

wherein

said first, second, and third program instructions are recorded on said medium.