Enforcing expected control flow in program execution

• US 8,645,923 B1
• Filed: 10/31/2008
• Issued: 02/04/2014
• Est. Priority Date: 10/31/2008
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Satish, Sourabh, McCorkendale, Bruce, Sobel, William E.
• Primary Examiner(s): Dao, Thuy
• Assistant Examiner(s): Chen, Xi D
• Application Number:
  US12/263,362
• Time in Patent Office: 1,922 Days
• Field of Search: None
• US Class Current: 717/131
• CPC Class Codes: G06F 21/52 during program execution, e.g. stack integrity ; Preventing unwanted data erasure; Buffer overflow
• CPC Class Codes: G06F 9/44589 Program code verification, e.g. Java bytecode verification, proof-carrying code high-level semantic checks G06F8/43; testing and debugging software G06F11/36

Abstract

When a program is loaded for execution, all code pages of the program except the one containing the entry point are set to be non-executable. When the executing program attempts to jump between code pages, an exception is thrown. Responsive to such an exception, a control flow graph of the program is examined, to determine if the attempted jump between code pages is expected. If the attempted jump is not expected, it is determined that the program is attempting a malicious activity. If the attempted jump is expected, the code page to which the program is attempting to jump is set to be executable, and control is returned to the program such that the jump executes.

Claims

1. A computer implemented method for enforcing control flow in an execution of a program, the method comprising the steps of:

preventing, by a computer, jumps between code pages in the execution of a program until verified against a control flow graph by setting all code pages of the program except a code page containing an entry point to be non-executable;

causing, by a computer, an exception handler to process an exception responsive to each attempted jump between code pages, and prior to execution of a destination code page;

responsive to an exception resulting from an attempted jump between code pages in the execution of the program, processing, by a computer, the exception by referring to the control flow graph concerning the program to determine whether the attempted jump between code pages is expected according to the control flow graph; and

responsive to at least whether the attempted jump between code pages is expected according to the control flow graph, determining, by a computer, whether the program is attempting a malicious action.

2. The method of claim 1 further comprising performing a at least one step from a group of steps consisting of:

receiving a control flow graph concerning the program;

building a control flow graph concerning the program;

building a control flow graph concerning the program, such that said control flow graph represents flow of control of the execution of a multi-threaded program on a per thread basis; and

building a control flow graph concerning the program, such that each node of said control flow graph comprises a code page of the program.

3. The method of claim 1 further comprising:

determining that the attempted jump between code pages is not expected according to the control flow graph;

responsive to determining that the attempted jump between code pages is not expected, determining that the program is attempting a malicious action; and

responsive to determining that the program is attempting a malicious action, performing at least one additional step from a group of steps consisting of;

not permitting the attempted jump to execute;

transmitting an alert to a central security service;

transmitting an alert to a user;

activating an anti-malware application;

terminating the program; and

modifying the program.

4. The method of claim 1 further comprising:

determining that the attempted jump between code pages is expected according to the control flow graph;

responsive to determining that the attempted jump between code pages is expected according to the control flow graph, determining that the program is not attempting a malicious action; and

responsive to determining that the program is not attempting a malicious action, allowing the jump to execute.

5. The method of claim 4 wherein allowing the jump to execute further comprises:

setting the code page to which the program is attempting to jump to be executable; and

returning control to the program such that the attempted jump executes.

6. The method of claim 4 wherein allowing the jump to execute further comprises:

modifying a use count associated with the code page to which the program is attempting to jump to indicate that a thread not currently executing in that code page is to execute therein; and

returning control to the program such that the attempted jump executes.

7. The method of claim 6 further comprising:

responsive to the use count associated with the code page to which the program is attempting to jump indicating that no threads are currently executing in that code page, setting that code page to be executable.

8. The method of claim 4 wherein allowing the jump to execute further comprises:

subsequently setting the code page from which the program is attempting to jump to be non-executable.

9. The method of claim 4 wherein allowing the jump to execute further comprises:

modifying a use count associated with the code page from which the program is attempting to jump to indicate that a thread currently executing in that code page is to cease executing therein.

10. The method of claim 9 further comprising:

responsive to the modified use count associated with the code page from which the program is attempting to jump indicating that no threads are currently executing in that code page, subsequently setting that code page to be non-executable.

11. The method of claim 1 further comprising:

monitoring code pages; and

setting at least one executable code page in which no thread is executing to be non-executable.

12. The method of claim 1 further comprising:

monitoring code pages; and

for at least one code page, modifying a use count associated with that code page to indicate that at least one thread has ceased executing in that code page.

13. The method of claim 12 further comprising:

responsive to a modified use count indicating that no threads are currently executing in the associated code page, setting that code page to be non-executable.

14. The method of claim 1 further comprising:

determining that at least one thread of the program has terminated; and

responsive to determining that the at least one thread has terminated, updating the control flow graph concerning the program to account for the termination of the at least one thread.

15. A system for enforcing control flow in an execution of a program, the system comprising:

hardware implemented control flow circuitry configured to prevent jumps between code pages in the execution of a program until verified against a control flow graph by setting all code pages of the program except a code page containing an entry point to be non-executable and cause an exception handler to process an exception responsive to each attempted jump between code pages, and prior to execution of a destination code page;

a set of hardware implemented registers configured to track an executable status of each code page of the program; and

hardware implemented exception handling circuitry, configured to process the exception by referring to a control flow graph concerning the program responsive to an exception resulting from an attempted jump between code pages in the execution of the program and to determine whether the attempted jump between code pages is expected according to the control flow graph;

wherein the exception handling circuitry is further configured, in response to determining that the attempted jump between code pages is not expected, to perform at least one additional step from a group of steps consisting of;

not permitting the attempted jump to execute, transmitting an alert to a central security service, transmitting an alert to a user, activating an anti-malware application, terminating the program, and modifying the program; and

wherein the exception handling circuitry is further configured, in response to determining that the attempted jump between code pages is expected, to allow the jump to execute by setting the code page to which the program is attempting to jump to be executable and returning control to the program such that the attempted jump executes.

16. The system of claim 15 further comprising:

hardware implemented control flow graphing circuitry, configured to build a control flow graph concerning the program.

17. The system of claim 16 further comprising:

at least one hardware implemented control flow graph register configured to store control flow graph data concerning the program.

18. The system of claim 16 wherein the control flow graphing circuitry is further configured to:

build the control flow graph concerning the program such that said control flow graph represents flow of control of the execution of a multi-threaded program on a per thread basis.

19. The system of claim 15 wherein the exception handling circuitry is further configured to execute at least one of the following steps as part of allowing the jump to execute:

modifying a use count associated with the code page to which the program is attempting to jump to indicate that a thread not currently executing in that code page is to execute therein; and

responsive to the use count associated with the code page to which the program is attempting to jump indicating that no threads are currently executing in that code page, setting that code page to be executable.

20. The system of claim 15 wherein the exception handling circuitry is further configured to execute at least one of the following steps as part of allowing the jump to execute:

subsequently setting the code page from which the program is attempting to jump to be non-executable;

modifying a use count associated with the code page from which the program is attempting to jump to indicate that a thread currently executing in that code page is to cease executing therein; and

responsive to the modified use count associated with the code page from which the program is attempting to jump indicating that no threads are currently executing in that code page, subsequently setting that code page to be non-executable.

21. The system of claim 15 further comprising hardware implemented code page monitoring circuitry configured to execute at least one of the following steps:

monitoring code pages;

setting at least one executable code page in which no thread is executing to be non-executable;

for at least one code page, modifying a use count associated with that code page to indicate that at least one thread has ceased executing in that code page; and

responsive to a modified use count indicating that no threads are currently executing in the associated code page, setting that code page to be non-executable.

22. The system of claim 15 wherein the exception handling circuitry is further configured to:

determine that at least one thread of the program has terminated, and responsive to determining that the at least one thread has terminated, to update the control flow graph concerning the program to account for the termination of the at least one thread.

23. At least one non-transitory computer readable medium storing a computer program product for enforcing control flow in an execution of a program, the computer program product comprising:

program code for preventing jumps between code pages in the execution of a program until verified against a control flow graph by setting all code pages of the program except a code page containing an entry point to be non-executable;

program code for causing, an exception handler to process an exception responsive to each attempted jump between code pages, and prior to execution of a destination code page;

program code for, responsive to an exception resulting from an attempted jump between code pages in the execution of the program, processing the exception by referring to the control flow graph concerning the program to determine whether the attempted jump between code pages is expected according to the control flow graph;

program code for, responsive to determining that the attempted jump between code pages is not expected, determining that the program is attempting a malicious action; and

program code for, responsive to determining that the attempted jump between code pages is expected, determining that the program is not attempting a malicious action and allowing the jump to execute.

24. The computer program product of claim 23 further comprising program code for performing at least one step from a group of steps consisting of:

receiving a control flow graph concerning the program;

building a control flow graph concerning the program;

building a control flow graph concerning the program, such that said control flow graph represents flow of control of the execution of a multi-threaded program on a per thread basis; and

building a control flow graph concerning the program, such that each node of said control flow graph comprises a code page of the program.

25. The computer program product of claim 23 further comprising program code for, responsive to determining that the program is attempting a malicious action, performing at least one additional step from a group of steps consisting of:

not permitting the attempted jump to execute;

transmitting an alert to a central security service;

transmitting an alert to a user;

activating an anti-malware application;

terminating the program; and

modifying the program.

26. The computer program product of claim 23 wherein the program code for allowing the jump to execute further comprises:

program code for setting the code page to which the program is attempting to jump to be executable; and

program code for returning control to the program such that the attempted jump executes.

27. The computer program product of claim 23 further comprising:

program code for modifying a use count associated with the code page to which the program is attempting to jump to indicate that a thread not currently executing in that code page is to execute therein;

program code for, responsive to the use count associated with the code page to which the program is attempting to jump indicating that no threads are currently executing in that code page, setting that code page to be executable; and

program code for returning control to the program such that the attempted jump executes.

28. The computer program product of claim 23 wherein the program code for allowing the jump to execute further comprises:

program code for subsequently setting the code page from which the program is attempting to jump to be non-executable.

29. The computer program product of claim 23 further comprising:

program code for modifying a use count associated with the code page from which the program is attempting to jump to indicate that a thread currently executing in that code page is to cease executing therein; and

program code for, responsive to the modified use count associated with the code page from which the program is attempting to jump indicating that no threads are currently executing in that code page, subsequently setting that code page to be non-executable.

30. The computer program product of claim 23 further comprising:

program code for monitoring code pages; and

program code for setting at least one executable code page in which no thread is executing to be non-executable.

31. The computer program product of claim 23 further comprising:

program code for monitoring code pages;

program code for, for at least one code page, modifying a use count associated with that code page to indicate that at least one thread has ceased executing in that code page; and

program code for, responsive to a modified use count indicating that no threads are currently executing in the associated code page, setting that code page to be non-executable.

32. The computer program product of claim 23 further comprising:

program code for determining that at least one thread of the program has terminated; and

program code for, responsive to determining that the at least one thread has terminated, updating the control flow graph concerning the program to account for the termination of the at least one thread.