System and methods for adaptive model generation for detecting intrusion in computer systems

• US 20130031633A1
• Filed: 09/10/2012
• Published: 01/31/2013
• Est. Priority Date: 01/25/2002
• Status: Active Grant

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Honig, Andrew, Howard, Andrew, Eskin, Eleazar, Stolfo, Salvatore J.
• Granted Patent: US 8,887,281 B2
• Time in Patent Office: Days
• US Class Current: 726/23
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/212 with details for data modelling support
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 21/554 involving event detection and direct action
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: H04L 63/14 for detecting or protecting against malicious traffic
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 Vulnerability analysis

Abstract

A system and methods for detecting intrusions in the operation of a computer system comprises a sensor configured to gather information regarding the operation of the computer system, to format the information in a data record having a predetermined format, and to transmit the data in the predetermined data format. A data warehouse is configured to receive the data record from the sensor in the predetermined data format and to store the data in a SQL database. A detection model generator is configured to request data records from the data warehouse in the predetermined data format, to generate an intrusion detection model based on said data records, and to transmit the intrusion detection model to the data warehouse according to the predetermined data format. A detector is configured to receive a data record in the predetermined data format from the sensor and to classify the data record in real-time as one of normal operation and an attack based on said intrusion detection model. A data analysis engine is configured to request data records from the data warehouse according to the predetermined data format and to perform a data processing function on the data records.

Claims

1. A system for detecting intrusions in the operation of a computer system comprising:

(a) a sensor configured to gather information regarding the operation of the computer system, to format the information in a data record, and to transmit the data record;

(b) one or more databases configured to receive the data record from the sensor, to store the data record, and to store an intrusion detection model;

(c) a detection model generator configured to request data records from the one or more databases, to generate the intrusion detection model based on said data records, and to transmit the intrusion detection model to the one or more databases;

(d) a detector configured to receive a data record from the sensor and to classify the data record in real-time as one of normal operation and an attack based on said intrusion detection model;

(e) a data analysis engine configured to request data records from the one or more databases and to perform a data processing function on the data records; and

(f) a detection model distributor configured to receive said intrusion detection model from the one or more databases and to transmit the detection model to at least one detector.

2. The system according to claim 1, wherein the data record is modified prior to storage in the one or more databases.

3. The system according to claim 1, wherein the sensor is a network sensor.

4. The system according to claim 1, wherein the sensor is a host sensor.

5. The system according to claim 1, wherein the data record comprises information regarding a source of the information.

6. The system according to claim 1, wherein the detection model generator is configured to periodically generate an updated intrusion detection model.

7. The system according to claim 1, wherein the detection model is a probabilistic model.

8. The system according to claim 7, wherein the detection model generator is configured to generate a parameterization of the probabilistic model.

9. The system according to claim 8, wherein the detector is configured to compute a probability associated with the data record.

10. The system according to claim 1, wherein the detection model is a set of support vectors which correspond to a decision boundary in a feature space.

11. The system according to claim 10, wherein the detection model generator is configured to generate a set of support vectors.

12. The system according to claim 11, wherein the detector is configured to map a data record to the feature space and determine the location of the data record in the feature space with respect to the decision boundary.

13. The system according to claim 1, wherein the sensor is configured to format the information in the data record in one or more predetermined formats.

14. The system according to claim 1, wherein the data analysis engine is further configured to append label data to the data records in the one or more databases.

15. The system according to claim 1, wherein the data analysis engine is configured to extract a feature from a plurality data records.

16. The system according to claim 15, wherein the data analysis engine is configured to append the feature data to the data records.

17. A method for detecting intrusions in the operation of a computer system comprising:

(a) gathering information regarding the operation of the computer system at a sensor and formatting the information into a data record;

(b) transmitting the data record to one or more databases, and storing the data record in the one or more databases;

(c) generating an intrusion detection model comprising requesting data records from the one or more databases, transmitting the intrusion detection model to the one or more databases, and storing the intrusion detection model at the one or more databases;

(d) classifying a data record as one of normal operation and an attack based on the intrusion detection model comprising receiving the data records from the sensor;

(e) requesting a data record from the one or more databases and performing a data processing function on the data record; and

(f) transmitting the intrusion detection model from a detection model distributor to at least one detector.

18. The method according to claim 17, further comprising modifying the data record prior storing the data record in the one or more databases.

19. The method according to claim 17, wherein the gathering information regarding the operation of the computer system at a sensor comprises providing a network sensor.

20. The method according to claim 17, wherein the gathering information regarding the operation of the computer system at a sensor comprises providing a host sensor.

21. The method according to claim 17, wherein the gathering information regarding the operation of the computer system at a sensor comprises formatting the data record with information relating to a source of the information.

22. The method according to claim 17, further comprising periodically generating the intrusion detection model.

23. The method according to claim 17, wherein the generating an intrusion detection model comprises generating a probabilistic model.

24. The method according to claim 23, wherein the classifying a data record as one of normal operation and an attack comprises computing a probability associated with the data record.

25. The method according to claim 17, wherein the generating an intrusion detection model comprises generating a set of support vectors which correspond to a decision boundary in a feature space.

26. The method according to claim 25, wherein the classifying a data record as one of normal operation and an attack comprises mapping a data record to the feature space and determining the location of the data record with respect to the decision boundary.

27. The method according to claim 17, wherein the formatting the information into a data record comprises formatting the information into the data record in one or more predetermined formats.

28. The method according to claim 17, comprising appending a label to the data records in the one or more databases.

29. The method according to claim 17, wherein the requesting data from the one or more databases comprises extracting a feature from a plurality data records.

30. The method according to claim 29, further comprising generating feature data based on the feature and appending the feature data to the data records in the one or more databases.