Detecting and preventing malcode execution

• US 7,971,255 B1
• Filed: 07/14/2005
• Issued: 06/28/2011
• Est. Priority Date: 07/15/2004
• Status: Expired due to Fees

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Kc, Gaurav S., Aho, Alfred V.
• Primary Examiner(s): Cervetti; David Garcia
• Application Number:
  US11/181,165
• Time in Patent Office: 2,175 Days
• Field of Search: 713/164, 713/188, 726/22, 726/23, 726/24, 726/25
• US Class Current: 726/24
• CPC Class Codes: G06F 12/1491 in a hierarchical protection system, e.g. privilege levels, memory rings
• CPC Class Codes: G06F 21/00 Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
• CPC Class Codes: G06F 2221/2101 Auditing as a secondary aspect
• CPC Class Codes: G06F 2221/2105 Dual mode as a secondary aspect
• CPC Class Codes: G06F 2221/2141 Access rights, e.g. capability lists, access control lists, access tables, access matrices
• CPC Class Codes: G06F 2221/2149 Restricted operating environment
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002

Abstract

A system for detecting and halting execution of malicious code includes a kernel-based system call interposition mechanism and a libc function interception mechanism. The kernel-based system call interposition mechanism detects a system call request from an application, determines a memory region from which the system call request emanates, and halts execution of the code responsible for the call request if the memory region from which the system call request emanates is a data memory region. The libc function interception mechanism maintains an alternative wrapper function for each of the relevant standard libc routines, intercepts a call from an application to one or more libc routines and redirects the call into the corresponding alternative wrapper function.

Claims

1. A monitoring system for detecting and halting execution of malicious code, the system comprising:

a processor that includes a kernel-based system call interposition mechanism and a standard library function interception mechanism, wherein the processor;

creates an alternative wrapper function that corresponds to one of a plurality of library functions in program code of an application, wherein the alternative wrapper function is interposed between the application and the plurality of library functions;

uses the alternative wrapper function to intercept a system call request from the application to a library function, verify whether return addresses associated with one or more intermediate functions associated with the system call request are located in write protected memory regions and verify a preceding instruction in the write protected memory region, and transmit a verification indication to an operating system kernel executing the system call request; and

uses the operating system kernel to execute the system call request based at least in part on the verification indication.

2. The monitoring system of claim 1, wherein the system call request includes a communication to the operating system kernel to request interaction with one or more resources.

3. The monitoring system of claim 1, wherein the processor is further configured to use the kernel-based system call interposition mechanism to set an indicator signaling presence and execution of malicious code if a memory region from which the system call request emanates is a data memory region.

4. The monitoring system of claim 1, wherein the processor is further configured to use the standard library function interception mechanism to (i) maintain the alternative wrapper function for each of the relevant standard C standard library routines, (ii) intercept a call from an application to one or more C standard library routines, and (iii) redirect the call into the corresponding alternative wrapper function.

5. The monitoring system of claim 4, wherein a stack walk is performed along a dynamic chain of each alternative wrapper function, so as to validate call site memory addresses and target site memory addresses against start limits and end limits, respectively, for C standard library routine associated with the alternative wrapper function.

6. The monitoring system of claim 5, wherein an innermost caller-callee pair in the dynamic chain is a direct call function.

7. The monitoring system of claim 1, further including a process enabling flag, wherein the kernel-based system call interposition mechanism and the standard library function interception mechanism are enabled to detect and halt execution of malicious code only when the process enabling flag is set.

8. The monitoring system of claim 1, wherein the system reports information about the malicious code upon detection of the malicious code.

9. A method, implemented on a processor, of detecting and halting execution of malicious code, the method comprising:

receiving system call invocations and invocations of C standard library functions at the processor;

creating an alternative wrapper function that corresponds to one of a plurality of library functions in program code of an application, wherein the alternative wrapper function is interposed between the application and the plurality of library functions;

using the alternative wrapper function to intercept a system call request from the application to a library function, verify whether return addresses associated with one or more intermediate functions associated with the system call request are located in write protected memory regions and verify a preceding instruction in the write protected memory region, and transmit a verification indication to an operating system kernel executing the system call request; and

using the operating system kernel to execute the system call request based at least in part on the verification indication.

10. The method of claim 9, further including halting execution of the code responsible if a memory region from which the system call request emanates includes writable sections of memory.

11. The method of claim 9, wherein the system call request includes a communication to the operating system kernel to request interaction with one or more resources.

12. The method of claim 9, further including setting an indicator signaling presence and execution of malicious code if a memory region from which the system call request emanates is a data memory region.

13. The method of claim 9, further including performing a stack walk along a dynamic chain of each alternative wrapper function, so as to validate call site memory addresses and target site memory addresses against start limits and end limits, respectively, for C standard library routine associated with the alternative wrapper function.

14. The method of claim 13, wherein an innermost caller-callee pair in the dynamic chain is a direct call function.

15. The method of claim 9, further including a process enabling flag, wherein the kernel-based system call interposition mechanism and the standard library function interception mechanism are enabled to detect and halt execution of malicious code only when the process enabling flag is set.

16. The method of claim 9, further including reporting information about the malicious code upon detection of the malicious code.