Secure firewall supporting different levels of authentication based on address or encryption status

• US 5,983,350 A
• Filed: 09/18/1996
• Issued: 11/09/1999
• Est. Priority Date: 09/18/1996
• Status: Expired due to Term

• Current Assignee: McAfee, LLC
• Original Assignee: Secure Computing Corporation (McAfee, LLC)
• Inventors: Minear, Spence, de Jongh, Troy, Stockwell, Edward B.
• Primary Examiner(s): Kim, Kenneth S.
• Application Number:
  US08/715,343
• Time in Patent Office: 1,147 Days
• Field of Search: 380/25, 380/42, 380/49, 395/187.01, 395/200.55, 395/200.6, 395/200.76, 395/200.79, 713/201, 709/225, 709/249
• US Class Current: 726/11
• CPC Class Codes: H04L 63/0227 Filtering policies mail message filtering H04L51/212
• CPC Class Codes: H04L 63/0485 Networking architectures for enhanced packet encryption processing, e.g. offloading of IPsec packet processing or efficient security association look-up
• CPC Class Codes: H04L 63/126 the source of the received data
• CPC Class Codes: H04L 63/164 at the network layer
• CPC Class Codes: H04L 69/16 Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
• CPC Class Codes: H04L 69/161 Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields

Abstract

A system and method for regulating the flow of messages through a firewall having a network protocol stack, wherein the network protocol stack includes an Internet Protocol (IP) layer, the method comprising establishing a security policy, determining, at the IP layer, if a message is encrypted, if the message is not encrypted, passing the unencrypted message up the network protocol stack to an application level proxy, and if the message is encrypted, decrypting the message and passing the decrypted message up the network protocol stack to the application level proxy, wherein decrypting the message includes executing a process at the IP layer to decrypt the message.

Claims

1. A method of regulating the flow of messages between an external network and an internal network through a firewall having a network protocol stack, wherein the network protocol stack includes an Internet Protocol (IP) layer, the method comprising:

establishing a security policy;

determining, at the IP layer, if a message to an IP address is encrypted;

if the message to the IP address is not encrypted, passing the unencrypted message up the network protocol stack to an application level proxy;

if the message to the IP address is encrypted, decrypting the message and passing the decrypted message up the network protocol stack to the application level proxy, wherein decrypting the message includes executing a procedure at the IP layer to decrypt the message;

determining at the application level proxy and based on the security policy if the message to that IP address is one that can be forwarded, wherein the decision whether to forward is a function of whether the message was encrypted when received; and

passing the message from the application level proxy to its destination through the IP layer.

2. A method of authenticating the sender of a message within a computer system having a network protocol stack, wherein the network protocol stack includes an Internet Protocol (IP) layer, the method comprising:

providing a plurality of authentication protocols, wherein each authentication protocol provides a different level of security;

determining, at the IP layer, if the message is encrypted;

if the message is encrypted, decrypting the message, wherein decrypting the message includes executing a process at the IP layer to decrypt the message;

passing the decrypted message up the network protocol stack to an application level proxy;

selecting an authentication protocol from the plurality of authentication protocols, wherein selecting includes determining an authentication protocol appropriate for the message, wherein the authentication protocol selected is a function of whether the message was encrypted when received;

executing, at the application level proxy, the authentication protocol to authenticate the sender of the message; and

passing the decrypted message through the IP layer to its destination.

3. The method according to claim 2 wherein determining an authentication protocol appropriate for the message includes:

determining a source IP address associated with the message; and

determining the authentication protocol associated with the source IP address.

4. A method of authenticating the sender of a message within a computer system having a network protocol stack, wherein the network protocol stack includes an Internet Protocol (IP) layer, the method comprising:

determining, at the IP layer, if the message is encrypted;

if the message is encrypted, decrypting the message, wherein decrypting the message includes executing a process at the IP layer to decrypt the message;

passing the decrypted message up the network protocol stack to an application level proxy;

determining an authentication protocol appropriate for the message wherein the message includes a security parameters index and wherein determining an authentication protocol appropriate for the message includes;

determining the authentication protocol associated with a dynamic IP address, wherein determining the authentication protocol includes looking up a security association based on the security parameters index;

determining a current address associated with the dynamic source IP address; and

binding the current address to the security parameters index;

executing, at the application level proxy, the authentication protocol to authenticate the sender of the message; and

passing the decrypted message through the IP layer to its destination.

5. The method according to claim 4 wherein the authentication protocol selected is a function of whether the message was encrypted when received.

6. A firewall, comprising:

a first communications interface;

a second communications interface;

a network protocol stack connected to the first and the second communications interfaces, wherein the network protocol stack includes an Internet Protocol (IP) layer and a transport layer;

a decryption procedure, operating at the IP layer, wherein the decryption procedure decrypts encrypted messages received at one of said first and second communications interfaces and outputs decrypted messages; and

an application layer proxy, connected to the transport layer of said network protocol stack, wherein the application layer proxy includes a plurality of authentication protocols, wherein each authentication protocol provides a different level of security, wherein the application layer proxy receives decrypted messages from the decryption procedure, selects an authentication protocol from the plurality of authentication protocols as a function of content of the decrypted message and whether the message was encrypted when received, executes the selected authentication protocol, and returns the message to the IP layer.

7. A firewall, comprising:

a first communications interface;

a second communications interface;

a first network protocol stack connected to the first communications interface, wherein the first network protocol stack includes an Internet Protocol (IP) layer and a transport layer;

a second network protocol stack connected to the second communications interface, wherein the second network protocol stack includes an Internet Protocol (IP) layer and a transport layer;

a security policy;

a decryption procedure, operating at the IP layer of the first network protocol stack, the decryption procedure receiving encrypted messages received by said first communications interface and outputting decrypted messages; and

an application layer proxy, connected to the transport layers of said first and second network protocol stacks, wherein the application layer proxy includes a plurality of authentication protocols, wherein each authentication protocol provides a different level of security, wherein the application layer proxy receives decrypted messages from the decryption procedure, selects an authentication protocol from the plurality of authentication protocols based on content of the decrypted message and whether the message was encrypted when received, and executes the selected authentication protocol; and

wherein the application layer proxy determines based on the security policy whether the message is to be forwarded, and wherein the message is returned to the IP layer if the message is to be forwarded.

8. A firewall, comprising:

a first communications interface;

a second communications interface;

a first network protocol stack connected to the first communications interface, wherein the first network protocol stack includes an Internet Protocol (IP) layer and a transport layer;

a second network protocol stack connected to the second communications interface, wherein the second network protocol stack includes an Internet Protocol (IP) layer and a transport layer;

a security policy;

a decryption procedure, operating at the IP layer of the first network protocol stack, the decryption procedure receiving encrypted messages received by said first communications interface and outputting decrypted messages; and

an application layer proxy, connected to the transport layers of said first and second network protocol stacks, wherein the application layer proxy includes a plurality of authentication protocols, wherein each authentication protocol provides a different level of security, wherein the application layer proxy receives decrypted messages from the decryption procedure, selects an authentication protocol from the plurality of authentication protocols based on the content of the decrypted message, and executes the selected authentication protocol and wherein the application layer proxy determines based on the security policy whether the message is to be forwarded, and wherein the message is returned to the IP layer if the message is to be forwarded;

a third communications interface; and

a third network protocol stack connected to the third communications interface and to the application layer proxy, wherein the third network protocol stack includes an Internet Protocol (IP) layer and a transport layer and wherein the second and third network protocol stacks are restricted to first and second burbs, respectively.

9. The firewall according to claim 8 wherein the authentication protocol is also based on whether the message was encrypted when received.

10. A method of establishing a virtual private network between a first and a second network, wherein each network includes an application level gateway firewall which uses a proxy operating at the application layer to process traffic through the firewall, wherein each firewall includes a network protocol stack and wherein each network protocol stack includes an Internet Protocol (IP) layer, the method comprising:

providing a plurality of authenication protocols, wherein each authentication prtocol provides a different level of security;

transferring a connection request from the first network to the second network;

determining, at the IP layer of the network protocol stack of the second network'"'"'s firewall, if the connection request is encrypted;

if the connection request is encrypted, decrypting the request, wherein decrypting the request includes executing a procedure at the IP layer of the second network'"'"'s firewall to decrypt the message;

passing the connection request up the network protocol stack to an application level proxy;

selecting an authentication protocol from the plurality of authentication protocols, wherein selecting includes determining an authentication protocol appropriate for the connection request, wherein the authentication protocol selected is a function of whether the message was encrypted when received;

executing the authentication protocol at the application level proxy to authenticate the connection request; and

if the connection request is authentic, establishing an active connection between the first and second networks and returning the connection request to the IP layer.

11. The method according to claim 10 wherein executing the authentication protocol includes executing program code within the firewall of the second network to mimic a challenge/response protocol executing on a server internal to the second network.

12. The method according to claim 10 wherein executing the authentication protocol includes executing program code to execute the authentication protocol in line to the session.

13. A computer-readable medium having computer-executable instructions for regulating the flow of messages between an external network and an internal network through a firewall having a network protocol stack, wherein the network protocol stack includes an Internet Protocol (IP), layer the instructions comprising:

instructions for establishing a security policy;

instructions for determining, at the IP layer, if a message to an IP address is encrypted;

instructions for passing the unencrypted message up the network protocol stack to an application level proxy if the message to the IP address is not encrypted;

instructions for decrypting the message and passing the decrypted message up the network protocol stack to the application level proxy if the message to the IP address is encrypted, wherein decrypting the message includes executing a procedure at the IP layer to decrypt the message;

instructions for determining at the application level proxy and based on the security policy if the message to that IP address is one that can be forwarded, wherein the decision whether to forward is a function of whether the message was encrypted when received; and

instructions for passing the message from the application level proxy to its destination through the IP layer.

14. A computer-readable medium having computer-executable instructions for authenticating the sender of a message within a computer system having a network protocol stack, wherein the network protocol stack includes an Internet Protocol (IP) layer the instructions comprising:

instructions for providing a plurality of authentication protocols, wherein each authentication protocol provides a different level of security;

instructions for determining, at the IP layer, if the message is encrypted;

instructions for decrypting the message if the message is encrypted, wherein decrypting the message includes executing a process at the IP layer to decrypt the message;

instructions for passing the decrypted message up the network protocol stack to an application level proxy;

instructions for selecting an authentication protocol from the plurality of authentication protocols, wherein selecting includes determining an authentication protocol appropriate for the message, wherein the authentication protocol selected is a function of whether the message was encrypted when received;

instructions for executing, at the application level proxy, the authentication protocol to authenticate the sender of the message; and

instructions for passing the decrypted message through the IP layer to its destination.

15. A computer-readable medium having computer-executable instructions for establishing a virtual private network between a first and a second network, wherein each network includes an application level gateway firewall which uses a proxy operating at the application layer to process traffic through the firewall, wherein each firewall includes a network protocol stack and wherein each network protocol stack includes an Internet Protocol (IP) layer, the instructions comprising:

instructions for providing a plurality of authentication protocols, wherein each authentication protocol provides a different level of security;

instructions for transferring a connection request from the first network to the second network;

instructions for determining, at the IP layer of the network protocol stack of the second network'"'"'s firewall, if the connection request is encrypted;

instructions for decrypting the request if encrypted, wherein decrypting the request includes executing a procedure at the IP layer of the second network'"'"'s firewall to decrypt the message;

instructions for passing the connection request up the network protocol stack to an application level proxy;

instructions for selecting an authentication protocol from the plurality of authentication protocols, wherein selecting includes determining an authentication protocol appropriate for the connection request, wherein the authentication protocol selected is a function of whether the message was encrypted when received;

instructions for executing the authentication protocol at the application level proxy to authenticate the connection request; and

instructions for establishing, if the connection request is authentic, an active connection between the first and second networks and for returning the connection request to the IP layer.

16. A computer-readable medium having computer-executable instructions for authenticating the sender of a message within a computer system having a network protocol stack, wherein the network protocol stack includes an Internet Protocol (IP) layer, the instructions comprising:

instructions for determining, at the IP layer, if the message is encrypted;

instructions for decrypting the message if encrypted, wherein decrypting the message includes executing a process at the IP layer to decrypt the message;

instructions for passing the decrypted message up the network protocol stack to an application level proxy;

instructions for determining an authentication protocol appropriate for the message wherein the message includes a security parameters index and wherein determining an authentication protocol appropriate for the message includes;

determining the authentication protocol associated with a dynamic IP address, wherein determining the authentication protocol includes looking up a security association based on the security parameters index;

determining a current address associated with the dynamic source IP address; and

binding the current address to the security parameters index;

instructions for executing, at the application level proxy, the authentication protocol to authenticate the sender of the message; and

instructions for passing the decrypted message through the IP layer to its destination.