Method for customizing processing and response for intrusion prevention

• US 7,657,937 B1
• Filed: 01/02/2004
• Issued: 02/02/2010
• Est. Priority Date: 01/02/2003
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Kumar, Dileep
• Primary Examiner(s): Dinh; Minh
• Application Number:
  US10/751,234
• Time in Patent Office: 2,223 Days
• Field of Search: None
• US Class Current: 726/23
• CPC Class Codes: H04L 63/0245 Filtering by information in the payload
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/166 at the transport layer

Abstract

A method for customizing the response for network based intrusion prevention comprising of: 1) virtual proxying the application data to enable custom response 2) enhancing transport layer (TCP/IP) to enable selective processing and selective modification of the stream for intrusion prevention. The invention also discloses a method for customizing the processing for both network or host based intrusion prevention comprising of: 1) loading externally defined processing procedures for the detection and prevention of intrusions 2) combining multiple of these processing procedures to form a unified processing engine that can be used for intrusion detection and prevention 3) unloading processing procedures that are not needed any more 4) loading new processing procedures that improve the intrusion detection and prevention.

Claims

1. A method for preventing intrusions to a computer system, comprising:

using a network-based appliance to intercept data packets;

deciding whether to forward the intercepted packets or whether to route the intercepted packets to a virtual proxy;

performing TCP or UDP processing on the intercepted packets before routing them to the virtual proxy;

using the virtual proxy to analyze the packets that have been routed to the virtual proxy to detect intrusions using a processing engine having at least one processing procedure that detects intrusions; and

when the virtual proxy detects an attack or violation in the packets, using the virtual proxy to direct a transport layer to modify the packets,

wherein the virtual proxy directs the transport layer to modify the packets using packet stream modification requests, the method further comprising sending the packet stream modification requests from the network-based appliance to a standby network-based appliance to support fault tolerance.

2. The method defined in claim 1 wherein using the virtual proxy to direct the transport layer to modify the packets further comprises:

modifying data in the packets at specified locations.

3. The method defined in claim 1 further comprising using the virtual proxy to direct the transport layer to remove data from the packets when the virtual proxy detects an attack or violation.