System and method for analyzing packed files

• US 8,578,495 B2
• Filed: 07/26/2006
• Issued: 11/05/2013
• Est. Priority Date: 07/26/2006
• Status: Active Grant

• Current Assignee: Carbonite LLC (Open Text Corporation)
• Original Assignee: Webroot Inc. (Open Text Corporation)
• Inventors: Burtscher, Michael
• Primary Examiner(s): Orgad, Edan
• Assistant Examiner(s): OLION, BRIAN L
• Application Number:
  US11/460,032
• Publication Number:
  US 20080028388A1
• Time in Patent Office: 2,659 Days
• Field of Search: 726/22, 726/24, 719331-332, 719/310
• US Class Current: 726/24
• CPC Class Codes: G06F 21/51 at application loading time, e.g. accepting, rejecting, starting or inhibiting executable software based on integrity or source reliability
• CPC Class Codes: G06F 21/54 by adding security routines or objects to programs
• CPC Class Codes: G06F 21/562 Static detection
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
• CPC Class Codes: G06F 9/445 Program loading or initiating bootstrapping G06F9/4401; security arrangements for program loading or initiating G06F21/57
• CPC Class Codes: G06F 9/44521 Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading

Abstract

A system and method for analyzing executable files on a computer is described. The method in one embodiment includes initiating, with an operating system of the computer, execution of a loader-process; loading, using the loader-process, code of a first executable file into an executable-memory of the computer; and executing the code of the first executable file, wherein the code of the first executable file unpacks other packed-code to generate unpacked code. In addition, the loader-process executes the unpacked code and stops execution of the unpacked code in response to the unpacked code attempting to make a potentially dangerous system call. The unpacked code is analyzed, in response to the unpacked code attempting to make the potentially dangerous system call, to assess whether the first executable file is a pestware file.

Claims

1. A method for analyzing executable files on a computer, comprising:

initiating, with an operating system of the computer, execution of a loader-process, wherein the loader-process has a memory space controlled by the loader-process;

loading, using the loader-process, code of a first executable file into the memory space;

executing the code of the first executable file in the memory space of the loader-process, wherein the code of the first executable file unpacks other packed code to generate unpacked code, the unpacked code including at least one system call;

analyzing the unpacked code to assess whether the first executable file is a pestware file;

enumerating dynamic link libraries (DLLS) that have been loaded by the operating system for the loader-process;

patching at least one export address table of a dynamic link library (DLL) associated with the loader-process so that the patched export address table refers the unpacked code loaded by the loader-process back to code associated with the loader module instead of functions provided by the operating system;

patching at least a portion of a function that at least one export address table points to so that the patched function refers an attempted access by a pestware process back to the loader-process, wherein at least one or more of a patched function and a patched DLL points to a location outside of the loader-process but within the loader-process'"'"'s address space;

executing the unpacked code in the memory space of the loader-process;

routing, in response to determining the at least one system call is safe, the at least one system call of the unpacked code from the memory space of the loader-process to the operating system of the computer for execution while the unpacked code is executed by the loader-process; and

clearing the memory space of the loader-process while maintaining the loader-process in a memory of the computer.

2. The method of claim 1, further comprising:

stopping the execution of the unpacked code in response to a predetermined event.

3. The method of claim 2, wherein the predetermined event is an event selected from the group consisting of:

the unpacked code attempting to make a potentially dangerous system call, the unpacked code terminating, and the unpacked code timing out.

4. The method of claim 1, wherein the packed code is packed in accordance with a code-altering technique selected from the group consisting of:

encryption, packing algorithms, compression techniques, weak encryption and file repackaging.

5. The method of claim 1, further comprising:

altering the export address table so the export address table entries point to code of the loader-process that may stop execution of code loaded by the loader-process.

6. The method of claim 1, further comprising:

altering selected functions so the unpacked code is unable to access code of the selected functions.

7. The method of claim 6, wherein the altering includes placing a jump instruction in the selected functions that points to code of the loader-process so as to prevent a second executable file from accessing the functions.

8. The method of claim 1, further comprising:

implementing a detour function that enables the loader-process to make an API call to load the first executable file.

9. The method of claim 1, wherein the analyzing includes analyzing portions of the unpacked code at offsets from a reference point within the unpacked code.

17. The method of claim 1, further comprising:

determining whether to stop the pestware process, unload the pestware process, or refer the pestware process to the actual function depending on the type of the call the pestware process attempts to make.

10. A system for analyzing a packed file stored on a computer comprising:

a processor configured to execute a loader module;

a memory space controlled by the loader module, wherein the loader module cooperating with the memory space performs the following steps;

executes a file unpacker in the memory space of the loader module, wherein the file unpacker, when executed by the loader module, unpacks the packed file to generate unpacked code;

executes the unpacked code in the memory space of the loader module;

assesses a system call from the unpacked code and, if the system call is safe, routes the system call from the unpacked code from the memory space of the loader module to the operating system for execution while the unpacked code is executed by the loader module;

enumerates dynamic link libraries (DLLS) that have been loaded by the operating system for the loader-process;

patches at least one export address table of a dynamic link library (DLL) associated with the loader module so that the patched export address table refers the unpacked code loaded by the loader module back to code associated with the loader module instead of functions provided by the operating system;

patches at least a portion of a function that at least one export address table points to so that the patched function refers an attempted access by a pestware process back to the loader module, wherein at least one or more of a patched function and a patched DLL points to a location outside of the loader-process but within the loader-process'"'"'s address space; and

clears the memory space of the loader-process while maintaining the loader-process in a memory of the computer; and

a detection module that analyzes the unpacked code after the unpacked code is executed by the loader module.

11. The system of claim 10, wherein the loader module stops execution of the unpacked code in response to the unpacked code attempting to carry out particular instructions while executing.

12. The system of claim 10, wherein the loader module fills in an import address table of the file unpacker.

18. The system of claim 10, wherein the loader module determines whether to stop the pestware process, unload the pestware process, or refer the pestware process to the actual function depending on the type of the call the pestware process attempts to make.

13. A non-transitory, processor-readable medium including instructions for analyzing executable files on a computer, the instructions comprising instructions for:

initiating, with an operating system of the computer, execution of a loader-process, wherein the loader-process has an address space controlled by the loader process;

loading, using the loader-process, code of a first executable file into the related address space of the loader-process;

executing the code of the first executable file in the address space of the loader-process, wherein the code of the first executable file unpacks other packed-code to generate unpacked code, the unpacked code including at least one system call;

analyzing the unpacked code, in response to the unpacked code attempting to make the at least one system call, to assess whether the first executable file is a pestware file;

enumerating dynamic link libraries (DLLS) that have been loaded by the operating system for the loader-process;

patching at least one export address table of a dynamic link library (DLL) associated with the loader-process so that the patched export address table refers the unpacked code loaded by the loader-process back to code associated with the loader module instead of functions provided by the operating system;

patching at least a portion of a function that at least one export address table points to so that the patched function refers an attempted access by a pestware process back to the loader-process, wherein at least one or more of a patched function and a patched DLL points to a location outside of the loader-process but within the loader-process'"'"'s address space;

executing the unpacked code in the address space of the loader-process;

routing, in response to determining the at least one system call is safe, the at least on system call of the unpacked code from the address space of the loader-process to the operating system of the computer for execution while the unpacked code is executed by the loader-process; and

clearing the memory space of the loader-process while maintaining the loader-process in a memory of the computer.

14. The non-transitory, processor-readable medium of claim 13, further comprising instructions for stopping execution of the unpacked code in response to a predetermined event selected from the group consisting of:

the unpacked code attempting to make a potentially dangerous system call, the unpacked code terminating, and the unpacked code timing out.

15. The non-transitory, processor-readable medium of claim 13, wherein the packed code is packed in accordance with a code-altering technique selected from the group consisting of:

encryption, packing algorithms, compression techniques, weak encryption and file repackaging.

16. The non-transitory, processor-readable medium of claim 13, further comprising instructions for altering selected functions so the unpacked code is unable to access code of the selected functions.

19. The non-transitory, processor-readable medium of claim 13, further comprising instructions for determining whether to stop the pestware process, unload the pestware process, or refer the pestware process to the actual function depending on the type of the call the pestware process attempts to make.