Dynamic software system intrusion detection

• US 6,681,331 B1
• Filed: 05/11/1999
• Issued: 01/20/2004
• Est. Priority Date: 05/11/1999
• Status: Expired due to Fees

• Current Assignee: StrataCloud, Inc. (Exitlag LLC)
• Original Assignee: Cylant, Inc.
• Inventors: Elbaum, Sebastian G., Munson, John C.
• Primary Examiner(s): Sheikh, Ayaz
• Assistant Examiner(s): REVAK, CHRISTOPHER A
• Application Number:
  US09/309,755
• Time in Patent Office: 1,715 Days
• Field of Search: 713/200, 713/201, 709/223, 709/224, 706/15, 706/16, 706/21, 706/22, 706/45-48, 706/50, 706/61, 706/62
• US Class Current: 726/23
• CPC Class Codes: G06F 11/30 Monitoring
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 21/55 Detecting local intrusion or implementing counter-measures
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting

Abstract

A real-time approach for detecting aberrant modes of system behavior induced by abnormal and unauthorized system activities that are indicative of an intrusive, undesired access of the system. This detection methodology is based on behavioral information obtained from a suitably instrumented computer program as it is executing. The theoretical foundation for the present invention is founded on a study of the internal behavior of the software system. As a software system is executing, it expresses a set of its many functionalities as sequential events. Each of these functionalities has a characteristic set of modules that is executed to implement the functionality. These module sets execute with clearly defined and measurable execution profiles, which change as the executed functionalities change. Over time, the normal behavior of the system will be defined by the boundary of the profiles. An attempt to violate the security of the system will result in behavior that is outside the normal activity of the system and thus result in a perturbation of the system in a manner outside the scope of the normal profiles. Such violations are detected by an analysis and comparison of the profiles generated from an instrumented software system against a set of known intrusion profiles and a varying criterion level of potential new intrusion events.

Claims

1. A method for detecting an intrusive use of a computer system through the dynamic measurement of software activity, comprising the steps of:

(a) within an internal operating environment on the computer system, but not limited to an operating system, causing the computer system to monitor control transfer respectively to and possibly from each of a plurality of program modules executing on the computer system, and execution profile data, said execution profile data identifying the modules that have executed within a period of time;

(b) at least temporarily retaining the execution profile data;

(c) estimating a nominal computer system behavior as a function of said execution profile data, while no intrusive use of the computer system is occurring; and

(d) detecting at least a potential intrusive use, if a change in the execution profile data indicates computer system behavior that deviates substantially from the nominal computer system behavior.

2. The method of claim 1, further comprising the step of indicating that at least the potential intrusive use of a selected program running on the computer system has occurred by registering an alarm.

12. The method of claim 1, further comprising the steps of creating a database of execution profiles, each execution profile corresponding to a known intrusive use of the computer system, an execution profile for each new detected intrusive use being added to the database of execution profiles, if not already included therein.

13. The method of claim 1, further comprising the step of producing one of a plurality of different levels of alarms in response to a probability of an intrusive use of the computer system having been detected as indicated by changes in the dynamic measurements of program activity, wherein a higher level alarm is produced as the probability of an intrusive use increases.

14. The method of claim 1, wherein the internal operating environment comprises at least one of:

(a) an operating system executing on a web server;

(b) an operating system in which a database program is executed;

(c) an operating system shell; and

(d) a file management system.

3. A method for detecting an intrusive use of a computer system, comprising the steps of:

within an internal operating environment on the computer system, but not limited to an operating system, causing the computer system to monitor transfer of control to and from each of a plurality of program modules executing on the computer system, and producing execution profile data identifying the modules that have executed within a period of time;

at least temporarily retaining the execution profile data;

estimating a nominal computer system behavior as a function of said execution profile data, while no intrusive use of the computer system is occurring; and

detecting at least a potential intrusive use based on the execution profile data;

wherein the step of estimating the nominal computer system behavior comprises the steps of;

(a) periodically identifying all of the program modules that executed during a current interval;

(b) periodically determining a frequency with which each program module executed during the current interval;

(c) as a function of the program modules that executed during the current interval and the frequency with which each program module executed, determining an execution profile;

(d) at least temporarily retaining execution profiles for successive intervals.

4. The method of claim 3, further comprising the step of normalizing the frequency for each program module when determining each execution profile.

5. The method of claim 4, further comprising the steps of:

(a) producing an execution profile for each successive interval, said execution profile indicating frequencies with which control has been transferred between specific program modules during the interval, each such transfer of control comprising a module transition; and

(b) at least temporarily retaining execution profiles for successive intervals.

6. The method of claim 4, further comprising the steps of:

(a) periodically mapping a sequence of module transitions representing subtrees into function sequences by mapping program functionalities to specific program modules that have executed during the current interval, producing function sequences for the current interval; and

(b) at least temporarily retaining the function sequences for successive intervals.

7. The method of claim 6, further comprising the steps of comparing the nominal computer system behavior with the execution profile for the current interval to determine whether any substantial difference exists that exceeds a predetermined threshold, and if so determining that at least the potential intrusive use of the computer system has occurred.

8. The method of claim 7, wherein if a potential intrusive use of the computer system has been detected, further comprising the steps of:

(a) providing a database of normal profiles, each profile corresponding to a known normal usage of the computer system;

(b) comparing the execution profile for the current interval with the database of normal profiles to determine if a departure from the normal profiles exists; and

if so

(c) indicating that an intrusive use of the computer system has occurred.

9. The method of claim 7, wherein if a potential intrusive use of the computer system has been detected, further comprising the steps of:

(a) providing a database of normal profiles, each profile having an associated transition matrix and corresponding to a known normal use of the computer system;

(b) comparing the execution profile for the current interval with the database of intrusion profiles to determine if a difference with the execution profile associated with each of the normal profiles exists; and

if so

(c) indicating that an intrusive use of the computer system has occurred.

10. The method of claim 7, wherein if a potential intrusive use of the computer system has been detected, further comprising the steps of:

(a) providing a database of execution profiles, each execution profile having an associated functional profile and corresponding to a known intrusive use of the computer system;

(b) comparing the functional profile for the current interval with the database of normal profiles to determine if a difference exists with the functional profile associated with one of the intrusion profiles exists; and

if so

(c) indicating that an intrusive use of the computer system has occurred.

11. The method of claim 7, further comprising the step of controlling a duration of subsequent intervals as a function of a variance of said difference.

15. A method for detecting an intrusive or an anomalous use of a computer program executing on a computer through the dynamic measurement of the computer program, said computer program comprising a plurality of program modules, comprising the steps of:

(a) providing a call to a mapping module, at each entry point into a program module and before each return to a program module within the computer program;

(b) using the mapping module to monitor execution profiles of program modules in response to each call, producing program module sequence data that indicate a sequence of program modules executing during successive defined sampling intervals and a frequency with which each such program module executed during each sampling interval;

(c) at least temporarily storing the program module sequence data provided by the mapping module for successive sampling intervals;

(d) providing a nominal set of data that represent a substantially steady state behavior of the computer program while executing with no intrusive or anomalous use; and

(e) employing at least one of a plurality of detection modes that detect an intrusive or an anomalous use of the computer program as a function of both the program module sequence data and the nominal set of data, by determining if a level of internal activity of the computer program differs substantially from an accepted range of nominal activity by the computer program.

16. The method of claim 15, further comprising the step of normalizing the frequency with which each program module executes during each successive sampling interval, producing an execution profile for each successive sampling interval, one of said plurality of detection modes comprising the steps of comparing a current execution profile for a most recent sampling interval with the nominal set of data to determine if there is a substantial difference, and if so, indicating that an intrusive or anomalous use of the computer program may have occurred.

18. The method of claim 15, further comprising the step of mapping the sequence of program modules that have executed during each successive sampling interval into function sequences by mapping program functionalities to the program modules, producing functional profiles for the successive sampling intervals, one of said plurality of detection modes comprising the steps of comparing current functional profiles for a most recent sampling interval with the nominal set of data to determine if there is a substantial difference, and if so, indicating that an intrusive or anomalous use of the computer program may have occurred.

19. The method of claim 15, further comprising the steps of:

(a) providing known intrusion profile data indicating activity occurring during different intrusive or anomalous uses of the computer program;

(b) comparing a current execution profile for a most recent sampling interval derived from the program module sequence data with the known intrusion profile data; and

(c) if the current execution profile substantially matches any of the known intrusion profile data, positively determining that an intrusive or anomalous use of the computer program has occurred.

20. The method of claim 15, further comprising the step of raising an alarm if it appears that an intrusive or anomalous use of the computer program has occurred as indicated by a change in the execution profiles.

21. The method of claim 15, further comprising the step of raising one of a plurality of different levels of alarm, the level of alarm raised depending upon a probability that an intrusive or anomalous use of the computer program has occurred as indicated by a change in the execution profiles.

22. The method of claim 15, further comprising the step of varying a duration of the sampling interval as a function of an extent of computational activity occurring during current sampling intervals as indicated by a change in the execution profiles.

23. The method of claim 15, wherein the step of providing a call to a mapping module comprises the step of adding the calls to the computer program.

24. The method of claim 15, wherein the step of providing a call to a mapping module comprises the step of adding a hardware monitor that transmits the call to the mapping module upon detecting an entry into any of the plurality of program modules and upon detecting a return from any of the plurality of program modules.

25. A computer-readable memory medium on which are stored a plurality of computer-executable instructions for performing steps (a)-(e), as recited in claim 15.

26. A computer-readable memory medium on which are stored a plurality of computer-executable instructions for performing the steps of the method of claim 16.

28. A computer-readable memory medium on which are stored a plurality of computer-executable instructions for performing the steps of the method of claim 18.

29. A computer-readable memory medium on which are stored a plurality of computer-executable instructions for performing the steps of the method of claim 19.

17. A method for detecting an intrusive or an anomalous use of a computer program executing on a computer, said computer program comprising a plurality of program modules, comprising the steps of:

(a) providing calls to a mapping module, at prescribed entry and return points in a plurality of program modules within the computer program;

(b) using the mapping module to monitor execution profiles of program modules, and producing program module sequence data that indicate a sequence of program modules executing during successive defined sampling intervals and a frequency with which each such program module executed during each sampling interval;

(c) at least temporarily storing the program module sequence data provided by the mapping module for successive sampling intervals;

(d) providing a nominal set of data that represent a substantially steady state behavior of the computer program while executing with no intrusive or anomalous use;

(e) employing at least one of a plurality of detection modes that detect an intrusive or an anomalous use of the computer program as a function of both the program module sequence data and the nominal set of data, by determining if a level of internal activity of the computer program differs substantially from an accepted range of nominal activity by the computer program; and

further comprising the step of creating an execution profile for each sampling interval from the program module sequence data that indicates frequencies with which execution of the computer program caused a transition between different program modules during the successive sampling intervals, one of said plurality of detection modes comprising the steps of comparing a current transition matrix for a most recent sampling interval with the nominal set of data to determine if there is a substantial difference, and if so, indicating that an intrusive or anomalous use of the computer program may have occurred.

27. A computer-readable memory medium on which are stored a plurality of computer-executable instructions for performing the steps of the method of claim 17.

30. Apparatus on which a computer program is executable, said apparatus being capable of detecting an intrusive or anomalous use of the computer program, comprising:

(a) a processor; and

(b) a memory coupled to the processor, said memory storing machine instructions that when executed by the processor implement the computer program and further, implement a plurality of functions, including;

(i) providing a call to a mapping module, at each entry point into a program module and possibly before each return to a program module within the computer program;

(ii) using the mapping module to monitor transitions between program modules in response to each call, producing program module sequence data that indicate a sequence of program modules executing on the processor during successive defined sampling intervals and a frequency with which each such program module executed during each sampling interval;

(iii) at least temporarily storing the program module sequence data provided by the mapping module for successive sampling intervals in the memory;

(iv) providing a nominal set of data in the memory, said nominal set of data representing a substantially steady state behavior of the computer program while executing on the processor with no intrusive or anomalous use; and

(v) employing at least one of a plurality of detection modes that detect an intrusive or an anomalous use of the computer program as a function of both the program module sequence data and the nominal set of data, by determining if a level of internal activity of the computer program differs substantially from an accepted range of nominal activity by the computer program.

31. The apparatus of claim 30, wherein the machine instructions further cause the processor to normalize the frequency with which each program module executes during each successive sampling interval, producing an execution profile for each successive sampling interval, one of said plurality of detection modes including comparing a current execution profile for a most recent sampling interval with the nominal set of data to determine if there if a substantial difference, and if so, causing the processor to indicate that an intrusive or anomalous use of the computer program may have occurred.

33. The apparatus of claim 30, wherein the machine instructions further cause the processor to map the sequence of program modules that have been executed by the processor during each successive sampling interval into function sequences by mapping program functionalities to the program modules, producing functional sequences for the successive sampling intervals, one of said plurality of detection modes including comparing current functional sequences for a most recent sampling interval with the nominal set of data stored in the memory to determine if there is a substantial difference, and if so, causing the processor to indicate that an intrusive or anomalous use of the computer program may have occurred.

34. The apparatus of claim 30, wherein the machine instructions further cause the processor to:

(a) access known intrusion profile data stored in the memory, which indicate activity occurring during different intrusive or anomalous uses of the computer program;

(b) compare a current profile for a most recent sampling interval derived from the program module sequence data with the known intrusion profile data; and

(c) if the current profile substantially matches any of the known intrusion profile data, positively determine that an intrusive or anomalous use of the computer program has occurred.

35. The apparatus of claim 30, further comprising a display, wherein the machine instructions further cause the processor to display an alarm on the display if it appears that an intrusive or anomalous use of the computer program has occurred.

32. Apparatus on which a computer program is executable, said apparatus being capable of detecting an intrusive or anomalous use of the computer program, comprising:

(a) a processor; and

(b) a memory coupled to the processor said memory storing machine instructions that when executed by the processor implement the computer program and further, implement a plurality of functions, including;

(i) providing a call to a mapping module, at each entry point into a program module and possibly also before each return to a program module within the computer program;

(ii) using the mapping module to monitor transitions between programs modules in response to each control transition, producing program module sequence data that indicate a sequence of program modules executing on the processor during successive defined sampling intervals and a frequency with which each such program module executed during each sampling interval;

(iii) at least temporarily storing the program module sequence data provided by the mapping module for successive sampling intervals in the memory;

(iv) providing a nominal set of data in the memory, said nominal set of data representing a substantially steady state behavior for the computer program while executing on the processor with no intrusive or anomalous use; and

(v) employing at least one of a plurality of detection modes that detect an intrusive or an anomalous use of the computer program as a function of both the program module sequence data and the nominal set of data, by determining if a level of internal activity of the computer program differs substantially from an accepted range of nominal activity by the computer program;

wherein the machine instructions further cause the processor to create a transition matrix in the memory for each sampling interval from the program module sequence data, the transition matrix indicating frequencies with which execution of the computer program by the processor caused a transition between different program modules during the successive sampling intervals, one of said plurality of detection modes including comparing a current transition matrix for a most recent sampling interval with the nominal set of data to determine if there is a substantial difference, and if so, causing the processor to indicate that as intrusive or anomalous use of the computer program may have occurred.