System and methods for adaptive model generation for detecting intrusion in computer systems

• US 9,497,203 B2
• Filed: 10/08/2014
• Issued: 11/15/2016
• Est. Priority Date: 01/25/2002
• Status: Active Grant

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Honig, Andrew, Howard, Andrew, Eskin, Eleazar, Stolfo, Salvatore J.
• Primary Examiner(s): Gergiso, Techane
• Application Number:
  US14/509,208
• Publication Number:
  US 20150058994A1
• Time in Patent Office: 769 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/212 with details for data modelling support
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 21/554 involving event detection and direct action
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: H04L 63/14 for detecting or protecting against malicious traffic
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 Vulnerability analysis

Abstract

A system and methods for detecting intrusions in the operation of a computer system comprises a sensor configured to gather information regarding the operation of the computer system, to format the information in a data record having a predetermined format, and to transmit the data in the predetermined data format. A data warehouse is configured to receive the data record from the sensor in the predetermined data format and to store the data in a SQL database. A detection model generator is configured to request data records from the data warehouse in the predetermined data format, to generate an intrusion detection model based on said data records, and to transmit the intrusion detection model to the data warehouse according to the predetermined data format. A detector is configured to receive a data record in the predetermined data format from the sensor and to classify the data record in real-time as one of normal operation and an attack based on said intrusion detection model. A data analysis engine is configured to request data records from the data warehouse according to the predetermined data format and to perform a data processing function on the data records.

Claims

1. A system for detecting intrusions in the operation of a computer system comprising:

(a) a plurality of sensors, each sensor configured to gather information regarding the operation of the computer system, to format the information in a data record, and to transmit the data record;

(b) one or more databases configured to receive the data record from the sensor, to store the data record, and to store an intrusion detection model;

(c) a detection model generator configured to request training data from a plurality of data records from the one or more databases, said training data comprising data from at least two sensors, to generate the intrusion detection model based on said training data from a plurality of data records, and to transmit the intrusion detection model to the one or more databases;

(d) a data analysis engine configured to request a data record from the one or more databases and to perform a data processing function on the data record;

(e) the detection model generator further configured to update the intrusion detection model in real-time;

(f) a detection model distributor configured to receive said intrusion detection model from the one or more databases and to transmit the detection model to at least one detector;

(g) one or more detectors configured to receive a data record from the sensor and to determine in real-time whether said data record corresponds to an attack based on said intrusion detection model;

(h) a visualization analysis engine configured to;

display the requested data record from the one or more databases in real time;

enable a system administrator to identify suspicious activity, not automatically identified by the intrusion detection model, as an attack in real-time; and

update the intrusion detection model based on the suspicious activity identified by the system administrator; and

(i) a data labeling tool configured to label the requested data record from the one or more databases if the data record corresponds to an attack.

2. The system according to claim 1, wherein the data record is modified prior to storage in the one or more databases.

3. The system according to claim 1, wherein the sensor is a network sensor.

4. The system according to claim 1, wherein the sensor is a host sensor.

5. The system according to claim 1, wherein the data record comprises information regarding a source of the information.

6. The system according to claim 1, wherein the detection model is a probabilistic model.

7. The system according to claim 6, wherein the detection model generator is configured to generate a parameterization of the probabilistic model.

8. The system according to claim 7, wherein the detector is configured to compute a probability associated with the data record.

9. The system according to claim 1, wherein the detection model is a set of support vectors which correspond to a decision boundary in a feature space.

10. The system according to claim 9, wherein the detection model generator is configured to generate a set of support vectors.

11. The system according to claim 10, wherein the detector is configured to map a data record to the feature space and determine the location of the data record in the feature space with respect to the decision boundary.

12. The system according to claim 1, wherein the sensor is configured to format the information in the data record in one or more predetermined formats.

13. The system according to claim 1, wherein the data analysis engine is further configured to append label data to the data records in the one or more databases.

14. The system according to claim 1, wherein the data analysis engine is configured to extract a feature from a plurality data records.

15. The system according to claim 14, wherein the data analysis engine is configured to append the feature data to the data records.

16. A method for detecting intrusions in the operation of a computer system comprising:

(a) gathering information regarding the operation of the computer system at a plurality of sensors and formatting the information from each sensor into a data record;

(b) transmitting the data record to one or more databases, and storing the data record in the one or more databases;

(c) generating an intrusion detection model comprising requesting training data from a plurality of data records from the one or more databases, said training data comprising data collected from at least two sensors, transmitting the intrusion detection model to the one or more databases, and storing the intrusion detection model at the one or more databases;

(d) requesting a data record from the one or more databases and performing a data processing function on the data record;

(e) updating the intrusion detection model in real-time;

(f) transmitting the intrusion detection model from a detection model distributor to at least one detector;

(g) determining in real-time whether a data record corresponds to an attack based on the intrusion detection model comprising receiving the data record from the sensor;

(h) displaying the requested data record from the one or more databases in real-time;

(i) enabling a system administrator to identify suspicious activity, not automatically identified by the intrusion detection model, as an attack in real-time;

(j) updating the intrusion detection model based on the suspicious activity identified by the system administrator; and

(k) labeling the requested data record from the one or more databases if the data record corresponds to an attack.

17. The method according to claim 16, further comprising modifying the data record prior to storing the data record in the one or more databases.

18. The method according to claim 16, wherein the gathering information regarding the operation of the computer system at a sensor comprises providing a network sensor.

19. The method according to claim 16, wherein the gathering information regarding the operation of the computer system at a sensor comprises providing a host sensor.

20. The method according to claim 16, wherein the gathering information regarding the operation of the computer system at a sensor comprises formatting the data record with information relating to a source of the information.

21. The method according to claim 16, wherein the generating an intrusion detection model comprises generating a probabilistic model.

22. The method according to claim 21, wherein the determining in real-time whether a data record corresponds to an attack comprises computing a probability associated with the data record.

23. The method according to claim 16, wherein the generating an intrusion detection model comprises generating a set of support vectors which correspond to a decision boundary in a feature space.

24. The method according to claim 23, wherein the determining in real-time whether a data record corresponds to an attack comprises mapping a data record to the feature space and determining the location of the data record with respect to the decision boundary.

25. The method according to claim 16, wherein the formatting the information into a data record comprises formatting the information into the data record in one or more predetermined formats.

26. The method according to claim 16, further comprising appending a label to the data records in the one or more databases.

27. The method according to claim 16, wherein the requesting data from the one or more databases comprises extracting a feature from a plurality data records.

28. The method according to claim 27, further comprising generating feature data based on the feature and appending the feature data to the data records in the one or more databases.