SMTP network security processing in a transparent relay in a computer network

• US 7,926,108 B2
• Filed: 11/17/2006
• Issued: 04/12/2011
• Est. Priority Date: 11/23/2005
• Status: Active Grant

• Current Assignee: Trend Micro Inc.
• Original Assignee: Trend Micro Inc.
• Inventors: Moriarty, Paul M., Rand, David L., Scharf, Gerald C., Esters, Scott D.
• Primary Examiner(s): Moazzami; Nasser
• Assistant Examiner(s): McNally; Michael S
• Application Number:
  US11/601,604
• Publication Number:
  US 20070204341A1
• Time in Patent Office: 1,607 Days
• Field of Search: 726/22
• US Class Current: 726/22
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: H04L 51/212 using filtering or selective blocking

Abstract

In one embodiment, a transparent relay receives diverted e-mail communications between an e-mail client and an e-mail server. The transparent relay may be configured to examine the e-mail communications for network security policy violations. E-mail communications that do not violate a network security policy may be relayed to their intended destination. Policy actions, such as discarding or redirection, may be performed on those that violate one or more network security policies. The transparent relay may include a pair of communications interfaces running in promiscuous mode, one for downstream communications and another for upstream communications. The transparent relay may decompose a network communication protocol to look network security policy violations.

Claims

1. A network security system for processing e-mail transactions, the system comprising:

an e-mail server on a first computer, the e-mail server being configured as a mail transfer agent;

an SMTP e-mail client on a second computer;

an SMTP transparent relay implemented separately from the first computer and the second computer, the SMTP transparent relay being configured to receive and process e-mail communications between the SMTP e-mail client and the e-mail server, the SMTP transparent relay being configured to examine the e-mail communications for network security policy violations, to perform policy actions on particular e-mail communications that violate a network security policy, and to relay particular e-mail communications that do not violate a network security policy; and

a router configured to divert to the SMTP transparent relay the e-mail communications between the SMTP e-mail client and the e-mail server, the e-mail communications having a destination IP address of the SMTP e-mail client or a destination IP address of the e-mail server as received by the router and as transmitted by the router to the SMTP transparent relay,

wherein the SMTP transparent relay includes a communications interface for each of the e-mail server and the SMTP e-mail client running in promiscuous mode.

2. The network security system of claim 1 wherein the SMTP transparent relay is configured to receive SMTP commands from the SMTP e-mail client and examine the SMTP commands for policy violations, perform policy actions on an SMTP command that violates a policy, and relay an SMTP command that do not violate a policy to the e-mail server.

3. The network security system of claim 1 wherein the SMTP transparent relay is configured to examine TCP packets initiating an e-mail connection between the SMTP e-mail client and the e-mail server for network security policy violations.

4. The network security system of claim 1 wherein the SMTP transparent relay is configured to examine SMTP responses from the e-mail server for network security policy violations.

5. The network security system of claim 1 wherein the policy actions include redirecting an e-mail communication to another computer.

6. A method of processing e-mail communications for network security, the method comprising:

transparently receiving in a computer configured as an SMTP transparent relay diverted e-mail packets originated by an SMTP e-mail client to be sent to an e-mail server, the e-mail server being configured as a mail transfer agent, each of the diverted e-mail packets having a destination IP address that does not correspond to any of the computer as received for processing by the computer, the SMTP transparent relay including a communication interface for each of the e-mail server and the SMTP client running in promiscuous mode;

in the computer, checking the diverted e-mail packets originated by the SMTP e-mail client for connection initiation packets configured to initiate an e-mail connection between the SMTP e-mail client and the e-mail server;

determining whether the connection initiation packets violate a first policy in a plurality of network security policies; and

performing a first policy action on the connection initiation packets if the connection initiation packets violate the first policy.

7. The method of claim 6 wherein the first policy action comprises redirecting the connection initiation packets to another computer configured to analyze the connection initiation packets for network security threats.

8. The method of claim 6 further comprising:

checking the diverted e-mail packets originated by the SMTP e-mail client for SMTP command packets;

determining whether the SMTP command packets violate a second policy in the plurality of network security policies; and

performing a second policy action on the SMTP command packets if the SMTP command packets violate the second policy.

9. The method of claim 8 wherein the second policy action comprises sending an error message to the e-mail client.

10. The method of claim 6 further comprising:

transparently receiving diverted e-mail packets originated by the e-mail server to be sent to the SMTP e-mail client;

checking the diverted e-mail packets originated by the e-mail server for SMTP response packets;

determining whether the SMTP response packets violate a third policy in the plurality of network security policies; and

performing a third policy action on the SMTP response packets if the SMTP response packets violate the third policy.

11. The method of claim 10 wherein the third policy comprises a prohibition against malformed SMTP response packets to block covert channels and the third policy action comprises blocking the SMTP response packet.

12. A method of processing computer communications for network security, the method comprising:

transparently receiving in a computer configured as an SMTP transparent relay diverted packets between a client computer configured as an SMTP e-mail client and a server computer configured as a mail transfer agent communicating over a communication session in accordance with SMTP, the diverted packets having a destination IP address of either the client computer or the server computer as received in the computer configured as the SMTP transparent relay, the computer configured as the SMTP transparent relay including a communications interface in promiscuous mode for each of the client computer and the server computer;

monitoring the communication session between the client computer and the server computer by checking SMTP commands sent by the client computer to the server computer and responses by the server computer to the SMTP commands sent by the client computer at different states of the SMTP to check for network security policy violations; and

relaying communications between the client computer and the server computer when the monitoring of the communication session does not indicate a network security policy violation.