Detecting a return-oriented programming exploit

• US 20120167120A1
• Filed: 12/22/2010
• Published: 06/28/2012
• Est. Priority Date: 12/22/2010
• Status: Active Grant

• Current Assignee: Withsecure Corporation
• Original Assignee: F-Secure Corporation (F-Secure Oyj)
• Inventors: Hentunen, Daavid
• Granted Patent: US 8,997,218 B2
• Time in Patent Office: Days
• US Class Current: 719/320
• CPC Class Codes: G06F 21/54 by adding security routines or objects to programs
• CPC Class Codes: G06F 21/577 Assessing vulnerabilities and evaluating computer system security
• CPC Class Codes: G06F 2221/033 Test or assess software

Abstract

A method and apparatus for detecting a Return-Oriented Programming exploitation. At a computer device, a mechanism to detect a control transfer of a code location in a memory is established. This may be, for example, hooking the control transfer. The code location relates to an electronic file. In the event that a control transfer of the code location is detected, a comparison is made between a destination code location address with values in the freed stack. If the code location address matches any of the values in the freed stack, then it is determined that the control transfer of the code location relates to a Return-Oriented Programming exploitation.

Claims

1. A method of detecting a Return-Oriented Programming exploitation of an application, the method comprising, at a computer device:

establishing a hooking rule to hook a code location relating to an electronic file stored in a computer readable medium in the form of a memory;

in the event that a control transfer of a code location relating to the electronic file is detected, comparing a code location address with values in the stack space freed by the control transfer and, in the event that the code location address and any of the values in the freed stack match, determining that the control transfer relates to a Return-Oriented Programming exploitation.

2. The method according to claim 1, wherein a function has a stack pointer, and the hooking is performed so as maintain the same value for the stack pointer.

3. The method according to claim 1, wherein the comparison of the code location address is made with one of a location address for a first value in the freed stack space and a location address for a second value in the freed stack space.

4. The method according to claim 1, wherein the comparison of the code location address is made with one of a number of location addresses for values in the freed stack space.

5. The method according to claim 4, further comprising:

estimating a number of location addresses to be compared with the return address.

6. The method according to claim 5, wherein the estimation is performed by:

determining a set of electronic files to inspect;

disassembling each electronic file to analyse functions related to each electronic file;

determining a number of parameters related to each function;

determining the function having the highest number of parameters; and

using the number of parameters for the function with the highest number of parameters as the estimate for the number of location addresses to be compared with the return address.

7. The method according to claim 5, wherein the estimation is performed by:

determining a set of electronic files to inspect;

disassembling each electronic file to analyse functions related to each electronic file;

disassembling each function to find return instructions and associated parameters;

determining the function having the highest number of parameters associated with return instructions; and

using the number of parameters for the function having the highest number of parameters associated with return instructions as the estimate for the number of location addresses to be compared with the return address.

8. The method according to claim 1, further comprising, prior to establishing a hooking rule to hook a function relating to an electronic file, determining that the electronic file does not use Address Space Layout Randomization.

9. The method according to claim 8, wherein the step of determining that an electronic file stored in a computer readable medium in the form of a memory does not use Address Space Layout Randomization comprises determining whether the electronic file was compiled using a /DYNAMICBASE linker option.

10. The method according to claim 1, further comprising taking action to prevent the Return-Oriented Programming exploitation

18. A computer program, comprising computer readable code which, when run on a computer device, causes the computer device to perform the method of claim 1.

11. A method of detecting a Return-Oriented Programming exploitation, the method comprising, at a computer device:

establishing a mechanism to detect a control transfer of a code location in a memory, the code location relating to an electronic file;

in the event that a control transfer of the code location is detected, comparing a destination code location address with values in the freed stack and, in the event that the code location address and any of the values in the freed stack match, determining that the control transfer of the code location relates to a Return-Oriented Programming exploitation.

12. The method according to claim 11, wherein the mechanism to detect a control transfer of a code location comprises:

marking the code location relating to the electronic file as not being present in the memory;

detecting an exception error when the control transfer attempts to transfer the code location to a memory location.

13. The method according to claim 11, wherein the mechanism to detect a control transfer of a code location comprises:

marking the code location relating to the electronic file as not being executable;

detecting an exception error when the control transfer attempts to transfer the code location to a memory location.

14. A computer device comprising:

a computer readable medium in the form of a memory;

at least one electronic file stored on the memory;

a hooking function for establishing a hooking rule to hook a code location relating to the electronic file;

a Return-Oriented Programming exploitation detection function for comparing a code location address with a value in the stack space freed by a control transfer and, in the event that the code location address and any of the values in the freed stack match, determining that the function relates to a Return-Oriented Programming exploitation.

15. The computer device according to claim 14, further comprising an application checking function for determining that the electronic file does not use Address Space Layout Randomization.

16. The computer device according to claim 14, wherein the Return-Oriented Programming exploitation detection function is arranged to compare the return address with one of a location address for a first value in the freed stack space and a location address for a second value in the freed stack space.

17. The computer device according to claim 14, wherein the Return-Oriented Programming exploitation detection function is arranged to compare the return address with one of a number of location addresses for values in the freed stack space.

19. The computer programme according to claim 17 which, when run on the computer device, causes the computer device to determine, prior to establishing a hooking rule, that the electronic file does not use Address Space Layer Randomization.

20. A computer program product comprising a computer readable medium and a computer program according to claim 17, wherein the computer program is stored on the computer readable medium.