System and methods for adaptive model generation for detecting intrusions in computer systems

• US 7,225,343 B1
• Filed: 01/27/2003
• Issued: 05/29/2007
• Est. Priority Date: 01/25/2002
• Status: Expired due to Term

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Honig, Andrew, Eskin, Eleazar, Stolfo, Salvatore J., Howard, Andrew
• Primary Examiner(s): Peeso; Thomas R.
• Application Number:
  US10/352,342
• Time in Patent Office: 1,583 Days
• Field of Search: 713/194, 713/193, 713/189
• US Class Current: 713/194
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/212 with details for data modelling support
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 21/554 involving event detection and direct action
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: H04L 63/14 for detecting or protecting against malicious traffic
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1433 Vulnerability analysis

Abstract

A system and methods for detecting intrusions in the operation of a computer system comprises a sensor configured to gather information regarding the operation of the computer system, to format the information in a data record having a predetermined format, and to transmit the data in the predetermined data format. A data warehouse is configured to receive the data record from the sensor in the predetermined data format and to store the data in a database. A detection model generator is configured to request data records from the data warehouse in the predetermined data format, to generate an intrusion detection model based on said data records, and to transmit the intrusion detection model to the data warehouse according to the predetermined data format. A detector is configured to receive a data record in the predetermined data format from the sensor and to classify the data record in real-time as one of normal operation and an attack based on said intrusion detection model. A data analysis engine is configured to request data records from the data warehouse according to the predetermined data format and to perform a data processing function on the data records.

Claims

1. A system for detecting intrusions in the operation of a computer system comprising:

(a) a sensor configured to gather information regarding the operation of the computer system, to format the information in a data record having a predetermined format, and to transmit the data in the predetermined data format;

(b) a data warehouse configured to receive the data record from the sensor in the predetermined data format, aggregate the data, store the data in a SQL database, and to store an intrusion detection model;

(c) a detection model generator configured to request data records from the data warehouse in the predetermined data format, to generate the intrusion detection model based on said data records, and to transmit the intrusion detection model to the data warehouse according to the predetermined data format;

(d) a detector configured to receive a data record in the predetermined data format from the sensor and to classify the data record in real-time as one of normal operation and an attack based on said intrusion detection model; and

(e) a data analysis engine configured to request data records from the data warehouse according to the predetermined data format and to perform a data processing function on the data records.

2. The system according to claim 1, wherein the sensor is a network sensor.

3. The system according to claim 1, wherein the sensor is a host sensor.

4. The system according to claim 1, wherein the data record comprises information regarding a source of the information.

5. A system for detecting intrusions in the operation of a computer system comprising:

(a) a sensor configured to gather information regarding the operation of the computer system, to format the information in a data record having a predetermined format, and to transmit the data in the predetermined data format;

(b) a data warehouse configured to receive the data record from the sensor in the predetermined data format and to store the data in a SQL database;

(c) a detection model generator configured to request data records from the data warehouse in the predetermined data format, to generate an intrusion detection model based on said data records, and to transmit the intrusion detection model to the data warehouse according to the predetermined data format;

(d) a detector configured to receive a data record in the predetermined data format from the sensor and to classify the data record in real-time as one of normal operation and an attack based on said intrusion detection model; and

(e) a data analysis engine configured to request data records from the data warehouse according to the predetermined data format and to perform a data processing function on the data records,

wherein the data analysis engine is configured to label a data record as one of normal operation and an attack.

6. The system according to claim 5, wherein the predetermined data format is XML markup language.

7. The system according to claim 5, wherein the detection model generator is configured to periodically generate the detection model, the system further comprising a detection model distributor configured to receive the detection model from the data warehouse and to transmit the detection model to the detector.

8. The system according to claim 5, wherein the detection model is a probabilistic model.

9. The system according to claim 8, wherein the detection model generator is configured to generate a parametrization of the probabilistic model.

10. The system according to claim 9, wherein the detector is configured to compute a probability associated with the data record.

11. The system according to claim 5 wherein the detection model is a set of support vectors which correspond to a decision boundary in a feature space.

12. The system according to claim 11, wherein the detection model generator is configured to generate a set of support vectors.

13. The system according to claim 12, wherein the detector is configured to map a data record to the feature space and determine the location of the data record in the feature space with respect to the decision boundary.

14. The system according to claim 5, wherein the data analysis engine is configured to display the data in the data warehouse as selected by a request in the predetermined data format.

15. The system according to claim 5, wherein the data analysis engine is further configured to append the label data to the data records in the SQL database.

16. The system according to claim 5, wherein the data analysis engine is configured to extract a feature from a plurality data records.

17. The system according to claim 16, wherein the data analysis engine is configured to append the feature data to the data records in the SQL database.

18. A distributed system for detecting intrusions in the operation of a computer system comprising:

(a) a plurality of sensors configured to gather information regarding the operation of the computer system, to format the information in a data record having a predetermined format, and to transmit the data in the predetermined data format;

(b) a data warehouse configured to receive the data record from the sensor in the predetermined data format, aggregate the data, store the data in a SQL database, and to store an intrusion detection model;

(c) a plurality of detection model generators configured to request data records from the data warehouse in the predetermined data format, each detection model generator configured to generate the respective intrusion detection model based on said data records, and to transmit the respective intrusion detection model to the data warehouse according to the predetermined data format;

(d) a plurality of detectors configured to receive a data record in the predetermined data format from the sensor and to classify the data record in real-time as one of normal operation and an attack based on a respective intrusion detection model; and

(e) a data analysis engine configured to request data records from the data warehouse according to the predetermined data format and to perform a data processing function on the data records.

19. The system according to claim 18, wherein the sensor is a network sensor.

20. The system according to claim 18, wherein the sensor is a host sensor.

21. The system according to claim 18, wherein the data record comprises information regarding a source of the information.

22. A distributed system for detecting intrusions in the operation of a computer system comprising:

(a) a plurality of sensors configured to gather information regarding the operation of the computer system, to format the information in a data record having a predetermined format, and to transmit the data in the predetermined data format;

(b) a data warehouse configured to receive the data record from the sensor in the predetermined data format and to store the data in a SQL database;

(c) a plurality of detection model generators configured to request data records from the data warehouse in the predetermined data format, each detection model generator configured to generate a respective intrusion detection model based on said data records, and to transmit the respective intrusion detection model to the data warehouse according to the predetermined data format;

(d) a plurality of detectors configured to receive a data record in the predetermined data format from the sensor and to classify the data record in real-time as one of normal operation and an attack based on a respective intrusion detection model; and

(e) a data analysis engine configured to request data records from the data warehouse according to the predetermined data format and to perform a data processing function on the data records,

wherein the data analysis engine is configured to label a data record as one of normal operation and an attack.

23. The system according to claim 22, wherein the predetermined data format is XML markup language.

24. The system according to claim 22, wherein one of the detection model generators is configured to periodically generate the respective detection model, the system further comprising a detection model distributor configured to receive the respective detection model from the data warehouse and to transmit the detection model to a respective detector.

25. The system according to claim 22, wherein one of the detection models is a probabilistic model.

26. The system according to claim 25, wherein one of the detection model generators is configured to generate a parametrization of the probabilistic model.

27. The system according to claim 26, wherein one of the detectors is configured to compute a probability associated with the data record.

28. The system according to claim 22, wherein one of the detection models is a set of support vectors which correspond to a decision boundary in a feature space.

29. The system according to claim 28, wherein one of the detection model generators is configured to generate a set of support vectors.

30. The system according to claim 29, wherein one of the detectors is configured to map a data record to the feature space and determine the location of the data record in the feature space with respect to the decision boundary.

31. The system according to claim 22, wherein the data analysis engine is configured to display the data in the data warehouse as selected by a request in the predetermined data format.

32. The system according to claim 22, wherein the data analysis engine is further configured to append the label data to the data records in the SQL database.

33. The system according to claim 22, wherein the data analysis engine is configured to extract a feature from a plurality data records.

34. The system according to claim 33, wherein the data analysis engine is configured to append the feature data to the data records in the SQL database.

35. A method for detecting intrusions in the operation of a computer system comprising:

(a) gathering information regarding the operation of the computer system at a sensor and formatting the information into a data record having a predetermined data format;

(b) transmitting the data record in the predetermined data format to a data warehouse, aggregating the data, and storing the data in a SQL database;

(c) generating an intrusion detection model comprising requesting data records from the data warehouse according to the predetermined data format, transmitting the intrusion detection model to the data warehouse in the predetermined data format, and storing the intrusion detection model at the data warehouse;

(d) classifying a data item in real-time as one of normal operation and an attack based on the intrusion detection model comprising receiving the data records from the sensor; and

(e) requesting a data record from the data warehouse according to the predetermined data format and performing a data processing function on the data record.

36. The method according to claim 35, wherein the step of gathering information regarding the operation of the computer system at a sensor comprises providing a network sensor.

37. The method according to claim 35, wherein the step of gathering information regarding the operation of the computer system at a sensor comprises providing a host sensor.

38. The method according to claim 35, wherein the step of gathering information regarding the operation of the computer system at a sensor comprises formatting the data record with information relating to a source of the information.

39. A method for detecting intrusions in the operation of a computer system comprising:

(a) gathering information regarding the operation of the computer system at a sensor and formatting the information into a data record having a predetermined data format;

(b) transmitting the data record in the predetermined data format to a data warehouse and storing the data in a SQL database;

(c) generating an intrusion detection model comprising requesting data records from the data warehouse according to the predetermined data format and transmitting the intrusion detection model to the data warehouse in the predetermined data format;

(d) classifying a data item in real-time as one of normal operation and an attack based on the intrusion detection model comprising receiving the data records from the sensor; and

(e) requesting a data record from the data warehouse according to the predetermined data format and performing a data processing function on the data record, wherein the step of requesting data from the data warehouse comprises labeling a data record as one of normal operation and an attack.

40. The method according to claim 39, wherein the step of formatting the data according to a predetermined data format comprises formatting the data according to XML markup language.

41. The system according to claim 39, further comprising-periodically generating the detection model and transmitting the detection model from the data warehouse to a distributor and transmitting the detection model from the distributor to the detector.

42. The system according to claim 39, wherein the step of generating an intrusion detection model comprises generating a probabilistic model.

43. The method according to claim 42, wherein the step of classifying a data record as normal operation or as an attack in real-time comprises computing a probability associated with the data record.

44. The system according to claim 39, wherein the step of generating an intrusion detection model comprises generating a set of support vectors in a feature space.

45. The method according to claim 44, wherein the step of classifying a data record as one of normal operation and an attack in real-time comprises mapping a data record to the feature space and determining the location of the data record with respect to the decision boundary.

46. The system according to claim 39, wherein the step of requesting data from the data warehouse comprises displaying the data record stored in the data warehouse as selected by a request according to the predetermined data format.

47. The method according to claim 39, comprising appending the label to the data records in the SQL database.

48. The system according to claim 39, wherein the step of requesting data from the data warehouse comprises extracting a feature from a plurality data records.

49. The method according to claim 48, further comprising appending the feature data to the data records in the SQL database.