Method of and system for detecting an anomalous operation of a computer system

• US 6,963,983 B2
• Filed: 01/13/2004
• Issued: 11/08/2005
• Est. Priority Date: 05/11/1999
• Status: Expired due to Fees

• Current Assignee: StrataCloud, Inc. (Exitlag LLC)
• Original Assignee: Cylant, Inc.
• Inventors: Elbaum, Sebastian G., Munson, John C.
• Primary Examiner(s): Revak, Christopher
• Application Number:
  US10/755,948
• Publication Number:
  US 20040143756A1
• Time in Patent Office: 665 Days
• Field of Search: 713/200, 713/201, 709/223, 709/224, 706/15, 706/16, 706/21, 706/22, 706/50, 706/61, 706 45- 48, 706/62
• US Class Current: 726/5
• CPC Class Codes: G06F 11/30 Monitoring
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 21/55 Detecting local intrusion or implementing counter-measures
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting

Abstract

A real-time approach for detecting aberrant modes of system behavior induced by abnormal and unauthorized system activities that are indicative of an intrusive, undesired access of the system. This detection methodology is based on behavioral information obtained from a suitably instrumented computer program as it is executing. The theoretical foundation for the present invention is founded on a study of the internal behavior of the software system. As a software system is executing, it expresses a set of its many functionalities as sequential events. Each of these functionalities has a characteristic set of modules that is executed to implement the functionality. These module sets execute with clearly defined and measurable execution profiles, which change as the executed functionalities change. Over time, the normal behavior of the system will be defined by the boundary of the profiles. An attempt to violate the security of the system will result in behavior that is outside the normal activity of the system and thus result in a perturbation of the system in a manner outside the scope of the normal profiles. Such violations are detected by an analysis and comparison of the profiles generated from an instrumented software system against a set of known intrusion profiles and a varying criterion level of potential new intrusion events.

Claims

1. A method of detecting an anomalous operation of a computer system indicative of a security violation, the method comprising:

(a) monitoring transitions across defined points within software executing on the computer system and in response thereto producing given program activity data;

(b) comparing the given program activity data with data indicative of a normal operation of the computer system to detect an anomalous operation of the computer system indicative of a security violation; and

(c) as a result of the comparison that detects the anomalous operation, taking a given action.

2. The method as described in claim 1 wherein the given action includes outputting a given notification.

3. The method as described in claim 2 wherein the given notification indicates an anomalous operation of the computer system.

4. The method as described in claim 1 wherein the given action includes updating data indicative of a normal operation of the software.

5. The method as described in claim 1 wherein the defined points within the software comprise program module input or output events.

6. The method as described in claim 1 wherein step (b) occurs at a given rate.

7. The method as described in claim 6 further including the step of adjusting the given rate.

8. The method as described in claim 7 wherein the given rate is increased during a given period of activity of the computer system.

9. The method as described in claim 7 wherein the given rate is decreased during a given period of activity of the computer system.

10. The method as described in claim 1 wherein the given action outputs a first indication if the comparison is indicative of an event that has been previously observed, and the given action outputs a second indication if the comparison is indicative of an event that has not been previously observed.

11. The method as described in claim 10 further including the step of forwarding data about the event that has not been previously observed for subsequent characterization.

12. The method as described in claim 1 further including the step of selectively displaying the given program activity data.

13. In a computer system comprising given hardware and software, the improvement comprising:

a transducer instrumented within the given hardware or the given software of the computer system that monitors the computer system as the computer system operates and in response thereto generates given program activity data;

a comparator that compares the given program activity data with data indicative of a normal operation of the computer system; and

a device for outputting a given indication based on the comparison between the given program activity data and the data indicative of the normal operation of the computer system;

wherein the given indication is indicative of an anomalous behavior resulting from a security violation in the computer system.

14. In the computer system as described in claim 13 wherein the transducer is given code that monitors transitions across defined points within the operating environment of the computer system.

15. In the computer system as described in claim 14 wherein the defined points within the operating environment of the computer system comprise program module input or output events.

16. In the computer system as described in claim 13 wherein the transducer obtains signals indicative of the operating environment of the computer system from a hardware bus.

17. In the computer system as described in claim 13 further including code for selectively adjusting a rate at which the comparator compares the given program activity data with data indicative of the normal operation of the computer system.

18. In the computer system as described in claim 13 further including a display for outputting the given program activity data.

19. In the computer system as described in claim 13 further including code for updating the data indicative of the normal operation of the computer system.

20. A computer system, comprising:

given hardware;

given software executable on the given hardware;

a transducer instrumented within the given hardware or the given software that monitors an operating environment of the computer system as the computer system operates and in response thereto generates given program execution trace data;

a comparator that compares the given program execution trace data with data indicative of a normal operation of the computer system; and

a device for outputting a given indication based on the comparison between the given program execution trace data and the data indicative of the normal operation of the computer system;

wherein the given indication is indicative of an anomalous behavior resulting from a security violation in the computer system.

21. A method of determining whether an intrusion has occurred at a given computer system having given hardware and given software, comprising:

instrumenting the given hardware or the given software;

instrumented hardware or software, monitoring an operating environment of the computer system as the computer system operates and in response thereto generating given program execution trace data;

comparing the given program execution trace data with data indicative of a normal operation of the computer system to determine whether an intrusion has occurred; and

based on the comparison that determines that an intrusion has occurred, taking a given action.

22. The method as described in claim 21 wherein the monitoring step is performed remotely from the computer system.

23. The method as described in claim 21 wherein the given action outputs an alarm.

24. The method as described in claim 21 wherein the given action updates the data indicative of the normal operation of the computer system.

25. The method as described in claim 21 further including the step of adjusting a rate at which the program execution trace data is compared with the data indicative of the nominal operation of the computer system as a function of an amount of processing activity taking place at the given computer system.

26. A method of detecting an anomalous operation of a computer system, indicative of a security violation, comprising:

establishing a steady state behavior of the computer system based on at least one execution profile;

comparing internally observable execution behavior of the computer system against the steady state behavior to detect an anomalous operation of the computer system indicative of the security violation, the internally observable execution behavior defined by program activity; and

taking a given action as a result of the comparison that detects the anomalous operation.

27. The method as described in claim 26 further including the step of adjusting a given rate of the comparison as a function of an amount of processing activity occurring in the computer system.

28. The method as described in claim 26 wherein the given action outputs an alarm.

29. The method as described in claim 28 wherein the alarm is indicative of an anomaly.

30. The method as described in claim 26 further including the step of recording the internally observable execution behavior of the computer system.

31. A method of detecting an anomalous operation of a computer system indicative of a security violation, comprising the unordered steps:

instrumenting given hardware or software in the computer system;

establishing a steady state behavior of the computer system based on at least one execution profile;

comparing a first behavior, as determined using program activity data generated from the instrumented hardware or software, against the steady state behavior to detect an anomalous operation of the computer system indicative of a security violation; and

taking a given action as a result of the comparison that indicates the anomalous operation.