Systems and methods for preventing exploitation of byte sequences that violate compiler-generated alignment

• US 8,434,073 B1
• Filed: 11/03/2008
• Issued: 04/30/2013
• Est. Priority Date: 11/03/2008
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Satish, Sourabh, McCorkendale, Bruce, Sobel, William E.
• Primary Examiner(s): Wong, Don
• Assistant Examiner(s): Rivera, Anibal
• Application Number:
  US12/263,739
• Time in Patent Office: 1,639 Days
• Field of Search: None
• US Class Current: 717/140
• CPC Class Codes: G06F 21/54 by adding security routines or objects to programs

Abstract

An exemplary method for preventing exploitation of byte sequences that violate compiler-generated instruction alignment may comprise: 1) identifying instantiation of a process, 2) identifying an address space associated with the process, 3) identifying, within the address space associated with the process, at least one control-transfer instruction, 4) determining that at least one byte preceding the control-transfer instruction is capable of resulting in an out-of-alignment instruction, and then 5) preventing the control-transfer instruction from being executed. In one example, the system may prevent the control-transfer instruction from being executed by inserting a hook in place of the intended instruction that executes the intended instruction and then returns control flow back to the instantiated process. Corresponding systems and computer-readable media are also disclosed.

Claims

1. A computer-implemented method for preventing exploitation of byte sequences that violate compiler-generated instruction alignment, the method comprising:

identifying instantiation of a process;

identifying an address space associated with the process;

identifying, within the address space associated with the process, at least one control-transfer instruction capable of directing control flow of the process;

determining that at least one byte that precedes the control-transfer instruction is capable of resulting in an out-of-alignment instruction that, when executed contrary to a compiler-generated instruction alignment for the process, results in at least one valid instruction;

preventing the control-transfer instruction from being executed contrary to the compiler-generated instruction alignment for the process by;

identifying an intended instruction within the process that contains the control-transfer instruction;

replacing the intended instruction with a hook that redirects control flow to a security patch that executes the intended instruction and then returns control flow back to the process;

upon replacing the intended instruction with the hook, allowing the process to execute.

2. The method of claim 1, wherein the security patch is located within the address space associated with the process.

3. The method of claim 1, wherein the security patch is located within an address space associated with a separate security process.

4. The method of claim 1, further comprising, prior to allowing the process to execute, determining whether the hook negatively affects intended control flow of the process.

5. The method of claim 1, wherein the out-of-alignment instruction comprises a portion of, but less than the entirety of, at least one of:

the intended instruction within the process;

two different intended instructions within the process.

6. The method of claim 1, wherein the control-transfer instruction comprises:

a return instruction;

a jump;

an interrupt;

a call.

7. The method of claim 1, wherein the out-of-alignment instruction comprises a system call.

8. The method of claim 1, wherein the control-transfer instruction comprises a portion of, but less than the entirety of, at least one of:

the intended instruction within the process;

two different intended instructions within the process.

9. The method of claim 1, wherein identifying the control-transfer instruction within the address space associated with the process comprises identifying each control-transfer instruction within the address space associated with the process.

10. A system for preventing exploitation of byte sequences that violate compiler-generated instruction alignment, the system comprising:

a process-analysis module programmed to;

identify instantiation of a process;

identify an address space associated with the process;

identify, within the address space associated with the process, at least one control-transfer instruction capable of directing control flow of the process;

determine that at least one byte that precedes the control-transfer instruction is capable of resulting in an out-of-alignment instruction that, when executed contrary to a compiler-generated instruction alignment for the process, results in at least one valid instruction;

a security module programmed to prevent the control-transfer instruction from being executed contrary to the compiler-generated instruction alignment for the process by;

identifying an intended instruction within the process that contains the control-transfer instruction;

replacing the intended instruction with a hook that redirects control flow to a security patch that executes the intended instruction and then returns control flow back to the process;

upon replacing the intended instruction with the hook, allowing the process to execute;

at least one processor configured to execute the process-analysis module and the security module.

11. The system of claim 10, wherein the security patch is located within the address space associated with the process.

12. The system of claim 10, wherein the security patch is located within an address space associated with a separate security process.

13. The system of claim 10, further comprising a testing module programmed to determine, prior to allowing the process to execute, whether the hook negatively affects intended control flow of the process.

14. The system of claim 10, wherein the out-of-alignment instruction comprises a portion of, but less than the entirety of, at least one of:

the intended instruction within the process;

two different intended instructions within the process.

15. The system of claim 10, wherein the control-transfer instruction comprises:

a return instruction;

a jump;

an interrupt;

a call.

16. The system of claim 10, wherein the out-of-alignment instruction comprises a system call.

17. The system of claim 10, wherein the control-transfer instruction comprises a portion of, but less than the entirety of, at least one of:

the intended instruction within the process;

two different intended instructions within the process.

18. A non-transitory computer-readable medium that, when executed by a computing device, causes the computing device to:

identify instantiation of a process;

identify an address space associated with the process;

identify, within the address space associated with the process, at least one control-transfer instruction capable of directing control flow of the process;

determine that at least one byte that precedes the control-transfer instruction is capable of resulting in an out-of-alignment instruction that, when executed contrary to a compiler-generated instruction alignment for the process, results in at least one valid instruction;

prevent the control-transfer instruction from being executed contrary to the compiler-generated instruction alignment for the process by;

identifying an intended instruction within the process that contains the control-transfer instruction;

replacing the intended instruction with a hook that redirects control flow to a security patch that executes the intended instruction and then returns control flow back to the process;

upon replacing the intended instruction with the hook, allow the process to execute.