SYSTEM AND METHOD FOR ANALYZING PACKED FILES

• US 20080028388A1
• Filed: 07/26/2006
• Published: 01/31/2008
• Est. Priority Date: 07/26/2006
• Status: Active Grant

• Current Assignee: Carbonite LLC (Open Text Corporation)
• Original Assignee: Webroot Inc. (Open Text Corporation)
• Inventors: Burtscher, Michael
• Granted Patent: US 8,578,495 B2
• Time in Patent Office: Days
• US Class Current: 717/174
• CPC Class Codes: G06F 21/51 at application loading time, e.g. accepting, rejecting, starting or inhibiting executable software based on integrity or source reliability
• CPC Class Codes: G06F 21/54 by adding security routines or objects to programs
• CPC Class Codes: G06F 21/562 Static detection
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
• CPC Class Codes: G06F 9/445 Program loading or initiating bootstrapping G06F9/4401; security arrangements for program loading or initiating G06F21/57
• CPC Class Codes: G06F 9/44521 Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading

Abstract

A system and method for analyzing executable files on a computer is described. The method in one embodiment includes initiating, with an operating system of the computer, execution of a loader-process; loading, using the loader-process, code of a first executable file into an executable-memory of the computer; and executing the code of the first executable file, wherein the code of the first executable file unpacks other packed-code to generate unpacked code. In addition, the loader-process executes the unpacked code and stops execution of the unpacked code in response to the unpacked code attempting to make a potentially dangerous system call. The unpacked code is analyzed, in response to the unpacked code attempting to make the potentially dangerous system call, to assess whether the first executable file is a pestware file.

Claims

1. A method for analyzing executable files on a computer, comprising:

initiating, with an operating system of the computer, execution of a loader-process;

loading, using the loader-process, code of a first executable file into an executable memory of the computer;

executing the code of the first executable file, wherein the code of the first executable file unpacks other packed-code to generate unpacked code; and

analyzing the unpacked code to assess whether the first executable file is a pestware file.

2. The method of claim 1, including:

executing the unpacked code; and

stopping the execution of the unpacked code in response to a predetermined event.

3. The method of claim 2, wherein the predetermined event is an event selected from the group consisting of:

the unpacked code attempting to make a potentially dangerous system call, the unpacked code terminating, and the unpacked code timing out.

4. The method of claim 1, wherein the packed code is packed in accordance with a code-altering technique selected from the group consisting of:

encryption, packing algorithms, compression techniques, weak encryption and file repackaging.

5. The method of claim 1 including:

routing, after determining a particular system call is safe, the particular system call to the operating system of the computer.

6. The method of claim 1 including:

enumerating dynamic link libraries (DLLS) that have been loaded by the operating system for the loader-process;

altering an export address table of at least one of the dynamic link libraries (DLLS) so that the export address table no longer points to a function entry point that the export address table originally pointed to.

7. The method of claim 6, wherein altering includes altering the export address table so the export address table entries point to code of the loader-process that may stop execution of code loaded by the loader-process.

8. The method of claim 1, including

altering selected functions so the unpacked code is unable to access code of the selected functions.

9. The method of claim 8, wherein the altering includes placing a jump instruction in the functions that points to code of the loader-process so as to prevent the second executable file from accessing the functions.

10. The method of claim 1, including

implementing a detour function that enables the loader-process to make an API call to load the first executable file.

11. The method of claim 1, wherein the analyzing includes analyzing portions of the unpacked code at offsets from a reference point within the unpacked code.

12. A system for analyzing a packed file stored on a computer comprising:

a loader module configured to execute a file unpacker, wherein the file unpacker, when executed by the loader module, unpacks the packed file to generate unpacked code; and

a detection module configured to analyze the unpacked code after the unpacked code is executed by the loader module.

13. The system of claim 12, wherein the loader module is configured to execute the unpacked code in a process space of the loader module.

14. The system of claim 13, wherein the loader module is configured to stop execution of the unpacked code in response to the unpacked code attempting to carry out particular instructions while executing.

15. The system of claim 12, wherein the loader module is configured to fill in an import address table of the file unpacker.

16. The system of claim 12, wherein the loader module is configured to patch at least one export address table of a dynamic link library (DLL) associated with the loader module so that the patched export address table refers the code loaded by the loader module back to code associated with the loader module.

17. The system of claim 16, wherein the loader module is configured to patch an exported function associated with the DLL so as to direct any calls from the unpacked code back to code associated with the loader module.

18. The system of claim 12, wherein the loader module is configured to assess a memory allocation call from the file unpacker, and if the call is safe, direct the call to the operating system.

19. A processor-readable medium including instructions for analyzing executable files on a computer, the instructions comprising instructions for:

initiating, with an operating system of the computer, execution of a loader-process;

loading, using the loader-process, code of a first executable file into an address space of the loader-process;

executing the code of the first executable file, wherein the code of the first executable file unpacks other packed-code to generate unpacked code; and

analyzing the unpacked code, in response to the unpacked code attempting to make the potentially dangerous system call, to assess whether the first executable file is a pestware file.

20. The processor-readable medium of claim 19 including instructions for;

executing the unpacked code; and

stopping execution of the unpacked code in response to a predetermined event selected from the group consisting of;

the unpacked code attempting to make a potentially dangerous system call, the unpacked code terminating, and the unpacked code timing out.

21. The processor-readable medium of claim 19, wherein the packed code is packed in accordance with a code-altering technique selected from the group consisting of:

encryption, packing algorithms, compression techniques, weak encryption and file repackaging.

22. The processor-readable medium of claim 20 including:

instructions for routing, after determining a particular system call is safe, the particular system call to the operating system of the computer.

23. The processor-readable medium of claim 19 including instructions for:

enumerating dynamic link libraries (DLLS) that have been loaded by the operating system for the loader-process; and

altering an export address table of at least one of the dynamic link libraries (DLLS) so that the export address table no longer points to a function.

24. The processor-readable medium of claim 20, wherein the instructions for altering include instructions for altering the export address table so the export address table points to code of the loader-process that stops execution of the unpacked code.

25. The processor-readable medium of claim 20, including instructions for altering selected functions so the unpacked code is unable to access code of the selected functions.