Method and system for filtering communications to prevent exploitation of a software vulnerability

• US 7,694,022 B2
• Filed: 09/30/2004
• Issued: 04/06/2010
• Est. Priority Date: 02/24/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation), Wolff Controls Corporation
• Original Assignee: Microsoft Corporation
• Inventors: Zugenmaier, Alf Peter, Guo, Chuanxiong, Simon, Daniel R., Garms, Jason, Wang, Jiahe Helen
• Primary Examiner(s): Vaughn, Jr.; William C
• Assistant Examiner(s): PEREZ, CARLOS R
• Application Number:
  US10/955,963
• Publication Number:
  US 20050198110A1
• Time in Patent Office: 2,014 Days
• Field of Search: 726 11- 14, 709/230, 709223-227, 714/4
• US Class Current: 709/249
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1433 Vulnerability analysis

Abstract

A method and system for protecting an application that implements a communication protocol against exploitation of a communication-based vulnerability is provided. A protection system provides a protection policy that specifies how to recognize messages that expose a specific vulnerability and specifies actions to take when the vulnerability is exposed. A protection policy specifies the sequence of messages and their payload characteristics that expose a vulnerability. The protection system may specify the sequences of messages using a message protocol state machine. A message protocol state machine of an application represents the states that the application transitions through as it receives various messages. The message protocol state machine of the protection policy may be a portion of the message protocol state machine of the application relating to the vulnerability. The protection system uses the message protocol state machine to track the states that lead up to the exposing of the vulnerability.

Claims

1. A method in a computer system with a processor and a memory for identifying when communications with a component exposes a vulnerability of the component, the component having a communication protocol with communication protocol states that the component transitions through as it receives communications, the method comprising:

providing a specification that specifies at least a portion of the communication protocol states of the component along with characteristics of communications associated with transitioning from one communication protocol state to another communication protocol state and indicates a communication protocol state when the vulnerability is exposed, the specification being independent of any exploitation of the vulnerability, an exploitation being a particular sequence of communications;

receiving a plurality of communications for the component; and

before each received communication is processed by the component,

determining a current communication protocol state of the component;

determining a next communication protocol state of the component based on the provided specification and the received message;

when the next communication protocol state is the indicated communication protocol state in which the vulnerability is exposed, not providing of the received communication to the component; and

when the next communication protocol state is not the indicated communication protocol state, providing the received communication to the component; and

setting the current protocol state to the next protocol state.

2. The method of claim 1 wherein the provided specification defines a state machine for controlling processing of communications.

3. The method of claim 2 wherein the state machine defines communication protocol states of the communication protocol and events to transition between communication protocol states.

4. The method of claim 2 wherein the specification specifies action to perform when transitioning between the communication protocol states.

5. The method of claim 4 wherein the determining is performed by a state machine engine that inputs the specification, a current communication protocol state, and a current event, selects a handler identified in the specification for the current communication protocol state and the current event and directs the execution of the handler to perform an action.

6. The method of claim 1 wherein the current communication protocol state is set on a per-component basis.

7. The method of claim 1 wherein the current communication protocol state is set on a per-session basis.

8. The method of claim 1 wherein the component handles multiple sessions and the determining of a communication protocol state is based only on communications of one session.

9. A system for detecting when a vulnerability of an application would be exposed as a result of messages being sent to the application, comprising:

a memory storing;

a specification for the application that defines a state machine that indicates when a message would expose the vulnerability of the application and defines actions to take to prevent exposing the vulnerability of the application, the state machine specifying communication protocol states that the application transitions through as it processes messages of a communication protocol used by the application, the specification being independent of any exploitation of the vulnerability, an exploitation being a particular series of messages;

a state store that stores a current communication protocol state of the application as represented by the state machine; and

an engine with computer executable instructions that receives messages for the application,

retrieves the current communication protocol state of the state machine for the application, and

identifies from the specification for the application actions to perform when in the retrieved current communication protocol state and that message is received; and

a processor for executing the computer-executable instructions stored in the memory.

10. The system of claim 9 wherein the state store stores state information on a per-session basis.

11. The system of claim 9 including a dispatcher that uses the specification to identify an application that is the target of a message based on a port identifier of the message.

12. The system of claim 9 including a dispatcher that identifies a session associated with a message based on a location of a session identifier within a message as defined by the specification for the application.

13. The system of claim 9 wherein when multiple vulnerabilities are defined for an application, a single specification is used to specify the vulnerabilities.

14. The system of claim 9 including a store that contains portions of a message until the entire message is received.

15. The system of claim 14 wherein the portions of the message are stored on a per-socket basis.

16. A computer-readable storage medium for controlling a computer system to identify when messages expose a vulnerability of an application, the application having a message protocol with message protocol states that the application transitions through as it receives messages, by a method comprising:

providing a specification that specifies at least a portion of the message protocol states of the application and indicates when, within the specified portion of the message protocol states, the vulnerability of the application is exposed, the provided specification defining a state machine for controlling processing of messages, the state machine defining message protocol states of the message protocol and events to transition between message protocol states, the specification being independent of any exploitation of the vulnerability, an exploitation being a particular sequence of messages;

receiving messages for the application;

determining whether a received message would expose the vulnerability of the application based on the specified portion of the message protocol states of the application, a current message protocol state, and the received message;

when it is determined that the vulnerability would be exposed, not providing the received message to the application; and

when it is determined that the vulnerability would be exposed, providing the received message to the application.