Enforcing the execution exception to prevent packers from evading the scanning of dynamically created code

• US 8,510,828 B1
• Filed: 12/31/2007
• Issued: 08/13/2013
• Est. Priority Date: 12/31/2007
• Status: Active Grant

• Current Assignee: NortonLifeLock Inc.
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Guo, Fanglu, Chiueh, Tzi-cker
• Primary Examiner(s): Arani, Taghi
• Assistant Examiner(s): VICTORIA, NARCISO F
• Application Number:
  US11/967,529
• Time in Patent Office: 2,052 Days
• Field of Search: 726/22
• US Class Current: 726/22
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities

Abstract

To detect possible malicious code that is unpacked at runtime before it is executed, antivirus software requires that any dynamically created code be scanned before it can be executed by a host computer system. This requirement may be enforced by requiring memory pages to be either executable or writable, but not both. Before changing from writable but not executable to executable but not writable, the page is scanned for malicious code. To prevent packers from evading this scanning, the software may enforce the execution exception to prevent packers from changing whether a page is executable and thereby evading the scanning of dynamically created code. The software may also include exception handlers to allow a program to write to a page that contains the code being executed, but also limit such an operation (e.g., to a single step) to avoid evasion of the antivirus software.

Claims

1. A computer program product for protecting a host computer system against dynamically unpacked malicious code, the computer program product comprising a non-transitory computer-readable storage medium containing executable computer program code for:

detecting an attempt by a program to write to a memory page on the host computer system;

responsive to detecting the attempt to write to the memory page, marking the memory page to be writable but non-executable to allow the program to write to the memory page;

detecting a request from the program to change the memory page to be executable;

preventing the memory page from being changed to be executable responsive to detecting the request from the program;

detecting an attempt by the program to execute code written to the memory page by the program, the execution attempt distinct from the request from the program to change the memory page to be executable;

responsive to detecting the attempt to execute code on the memory page, before executing the code on the memory page, scanning the memory page for malicious code; and

marking the memory page to be executable if the scanning does not find malicious code on the memory page.

2. The computer program product of claim 1, wherein preventing the memory page from being changed to be executable comprises:

after the memory page is changed to be executable, reversing the change to make the memory page non-executable.

3. The computer program product of claim 1, the non-transitory computer-readable storage medium further containing computer program code for:

recording an intended page protection for the memory page as being executable responsive to detecting the request from the program to change the memory page to be executable, although a page protection for the memory page is non-executable.

4. The computer program product of claim 3, the non-transitory computer-readable storage medium further containing computer program code for:

responsive to an inquiry from the program about the page protection of the memory page, responding to the program with the intended page protection of executable although the page protection for the memory page is non-executable.

5. The computer program product of claim 1, wherein detecting the attempt to write to the memory page comprises intercepting a first exception generated by a memory manager when the program attempts to write to the memory page, the memory page set to be executable but not writable.

6. The computer program product of claim 1, wherein detecting the attempt to execute code written to the memory page comprises intercepting a second exception generated by a memory manager when the program attempts to execute the code written to the memory page by the program, the memory page set to be writable but not executable.

7. The computer program product of claim 1, the non-transitory computer-readable storage medium further containing computer program code for:

keeping the memory page as being non-executable if the scanning identifies malicious code on the memory page.

8. The computer program product of claim 1, the non-transitory computer-readable storage medium further containing computer program code for:

declaring an existence of malicious code if the scanning identifies malicious code on the memory page.

9. The computer program product of claim 1, wherein scanning the memory page for malicious code comprises analyzing the memory page using a set of known signatures for malicious code.

10. A computer-implemented method for protecting a host computer system against dynamically unpacked malicious code, the method comprising:

detecting an attempt by a program to write to a memory page on the host computer system;

responsive to detecting the attempt to write to the memory page, marking the memory page to be writable but non-executable to allow the program to write to the memory page;

detecting a request from the program to change the memory page to be executable;

preventing the memory page from being changed to be executable responsive to detecting the request from the program;

detecting an attempt by the program to execute code written to the memory page by the program, the execution attempt distinct from the request from the program to change the memory page to be executable;

responsive to detecting the attempt to execute code on the memory page, before executing the code on the memory page, scanning the memory page for malicious code; and

marking the memory page to be executable if the scanning does not find malicious code on the memory page.

11. The computer-implemented method of claim 10, further comprising:

recording attempts by the program to change whether a portion of the memory page is executable.

12. The computer-implemented method of claim 10, wherein detecting the attempt by the program to write to the memory page comprises:

setting the memory page to be not writeable; and

intercepting an exception when the program attempts to write to the memory page.

13. The computer-implemented method of claim 10, further comprising:

keeping the memory page as being non-executable if the scanning identifies malicious code on the memory page.

14. The computer-implemented method of claim 10, further comprising:

declaring an existence of malicious code if the scanning identifies malicious code on the memory page.

15. The computer-implemented method of claim 10, wherein scanning the memory page for malicious code comprises analyzing the memory page using a set of known signatures for malicious code.

16. A computer program product for protecting a host computer system against dynamically unpacked malicious code, the computer program product comprising a non-transitory computer-readable storage medium containing executable computer program code for:

enforcing a mutual exclusion requirement for a set of memory pages on the host computer system, the mutual exclusion requirement requiring that each of the set of memory pages cannot be both executable and writeable at the same time;

executing a set of instructions for a program contained on a first memory page of the set of memory pages, a page protection of the first memory page set to be executable but not writeable, wherein the set of instructions includes a write instruction to write to the first memory page;

setting a single step execution before passing control to the program;

changing the page protection of the first memory page to be writeable and executable to allow the write instruction to be executed;

passing control to the program to execute the write instruction;

intercepting the single step execution after the write instruction is executed by the program; and

changing the page protection of the first memory page to be executable and not writable after the write instruction is executed.

17. The computer program product of claim 16, the non-transitory computer-readable storage medium further containing computer program code for:

scanning any dynamically created code on the memory pages before executing the dynamically created code.

18. The computer program product of claim 17, wherein scanning any dynamically created code on the memory pages before executing the dynamically created code comprises:

detecting an attempt to execute code on a particular memory page;

responsive to detecting the attempt to execute code on the memory page, before executing the code on the memory page, scanning the memory page for malicious code; and

marking the memory page to be executable if the scanning does not find malicious code on the memory page.

19. The computer program product of claim 18, wherein scanning the memory page for malicious code comprises analyzing the memory page using a set of known signatures for malicious code.