Dynamic replacement of system call tables

• US 7,685,638 B1
• Filed: 12/13/2005
• Issued: 03/23/2010
• Est. Priority Date: 12/13/2005
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Buches, David
• Primary Examiner(s): Vu; Kimyen
• Assistant Examiner(s): King; John B
• Application Number:
  US11/302,706
• Time in Patent Office: 1,561 Days
• Field of Search: 726 22- 25
• US Class Current: 726/22
• CPC Class Codes: G06F 21/53 by executing in a restricted environment, e.g. sandbox or secure virtual machine

Abstract

Techniques are disclosed that exploit system call mechanism to effect robust security applications. In one particular case, security software is able to effectively “sandbox” user mode applications at the thread granularity level, by replacing the system call mechanism of the operating system with a custom mechanism that limits the rights available to a target application that is vulnerable to malicious attack. The techniques allow the security software to create service tables with varying degrees of security levels, and do not impact performance of non-targeted running processes/threads.

Claims

1. A method for dynamically replacing a system call table for applications targeted for security checking, comprising:

identifying a target process executing on a processor including threads associated with a global service descriptor table (SDT);

allocating an alternate SDT for the target process;

installing one or more security hook routines in the alternate SDT;

changing the threads included in the target process from being associated with the global SDT to being associated with the alternate SDT, wherein threads of a plurality of other processes remain associated with the global SDT; and

responsive to a thread of the target process creating a new thread, associating the new thread with the alternate SDT.

2. The method of claim 1 wherein upon a call issued by a thread of the target process, the method further comprises executing one or more of the security hook routines of the alternate SDT.

3. The method of claim 2 wherein in response to determining that the one or more security hook routines have completed execution, the method further comprises transferring control to a native service of the target process, or back to the calling thread with an error status.

4. The method of claim 3 wherein in response to the error status indicating that hook routine security checking failed, the method continues with remedial processing.

5. The method of claim 4 wherein the remedial processing includes at least one of sending a notification of malicious activity, identification of the malicious activity, impeding the malicious activity, containing the malicious activity, and observing the malicious activity.

6. A machine-readable storage medium encoded with executable instructions, that when executed by a processor, cause the processor to carry out a process for dynamically replacing a system call table for applications targeted for security checking, the process comprising:

identifying a target process including threads associated with a global service descriptor table (SDT) object;

allocating an alternate SDT for the target process;

installing one or more security hook routines in the alternate SDT;

changing the threads included in the target process from being associated with the global SDT to being associated with the alternate SDT, wherein threads of a plurality of other processes remain associated with the global SDT; and

responsive to a thread of the target process creating a new thread, associating the new thread with the alternate SDT.

7. The machine-readable storage medium of claim 6 wherein upon a call issued by a thread of the target process, the process further comprises executing one or more of the security hook routines of the alternate SDT.

8. The machine-readable storage medium of claim 7 wherein in response to determining that the one or more security hook routines have completed execution, the process further comprises transferring control to a native service of the target process, or back to the calling thread with an error status.

9. The machine-readable storage medium of claim 8 wherein in response to the error status indicating that hook routine security checking failed, the process continues with remedial processing.

10. The machine-readable storage medium of claim 9 wherein the remedial processing includes at least one of sending a notification of malicious activity, identification of the malicious activity, impeding the malicious activity, containing the malicious activity, and observing the malicious activity.

11. A system for dynamically replacing a system call table for applications targeted for security checking, comprising:

a storage medium for storing one or more security hook routines; and

an alternate service descriptor table (SDT) module for;

identifying a target process including threads associated with a global SDT;

allocating an alternate SDT for the target process;

installing one or more of the security hook routines from the storage medium in the alternate SDT;

changing the threads included in the target process from being associated with the global SDT to being associated with the alternate SDT, wherein threads of a plurality of other processes remain associated with the global SDT; and

responsive to a thread of the target process creating a new thread, associating the new thread with the alternate SDT.

12. The system of claim 11 wherein upon a call issued by a thread of the target process, the system executes one or more of the security hook routines of the alternate SDT.

13. The system of claim 12 wherein in response to determining that the one or more security hook routines have completed execution, the system transfers control to a native service of the target process, or back to the calling thread with an error status.

14. The system of claim 13 wherein in response to the error status indicating that hook routine security checking failed, the system executes remedial processing.

15. The system of claim 14 wherein the remedial processing includes at least one of sending a notification of malicious activity, identification of the malicious activity, impeding the malicious activity, containing the malicious activity, and observing the malicious activity.

16. A system for dynamically replacing a system call table for applications targeted for security checking, comprising:

means for storing one or more security hook routines;

means for identifying a target process including threads associated with a global service descriptor table (SDT);

means for allocating an alternate SDT for the target process;

means for installing one or more of the security hook routines from the means for storing in the alternate SDT;

means for changing the threads included in the target process from being associated with the global SDT to being associated with the alternate SDT, wherein threads of a plurality of other processes remain associated with the global SDT; and

means for associating a new thread with the alternate SDT, responsive to a thread of the target process creating the new thread.

17. The system of claim 16 further comprising a means for executing one or more of the security hook routines of the alternate SDT, upon a call issued by a thread of the target process.