Detecting return-oriented programming payloads by evaluating data for a gadget address space address and determining whether operations associated with instructions beginning at the address indicate a return-oriented programming payload

• US 9,495,541 B2
• Filed: 09/17/2012
• Issued: 11/15/2016
• Est. Priority Date: 09/15/2011
• Status: Active Grant

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Polychronakis, Michalis, Keromytis, Angelos
• Primary Examiner(s): Patel, Haresh N
• Application Number:
  US14/344,458
• Publication Number:
  US 20140344932A1
• Time in Patent Office: 1,520 Days
• Field of Search: 726/23
• US Class Current: 1/1
• CPC Class Codes: G06F 21/54 by adding security routines or objects to programs
• CPC Class Codes: G06F 21/562 Static detection
• CPC Class Codes: G06F 2221/034 Test or assess a computer or a system

Abstract

Systems, methods, and media for detecting the presence of return-oriented programming (ROP) payloads are provided, comprising; identifying a potential gadget address space; determining if a piece of the data corresponds to an address of the potential gadget address space; and in response to determining that the piece of the data corresponds to an address of the potential gadget address space: determining whether a plurality of operations, each associated one of a plurality instructions beginning at the address, indicates that an ROP payload is present in the data, and indicating that an ROP payload is present in the data in response to making a determination that a plurality of operations indicates that an ROP payload is present in the data a given number of times.

Claims

1. A system for detecting the presence of a return-oriented programming (ROP) payload in data, comprising:

a hardware processor that;

identifies a potential gadget address space;

initializes a virtual address space of an emulator with instructions from the potential gadget address space;

determines if a piece of the data corresponds to an address of the potential gadget address space; and

in response to determining that the piece of the data corresponds to an address of the potential gadget address space;

for each instruction of a plurality of instructions beginning at the address;

determines whether the instruction is valid;

counts the instruction as part of an instruction count; and

determines whether the instruction count meets at least one threshold;

in response to determining that one of the plurality of instructions is valid and determining that the instruction count meets the at least one threshold, increases a gadget count; and

indicates that an ROP payload is present in the data in response to the gadget count meeting a threshold greater than one.

2. The system of claim 1, wherein in determining whether the instruction is valid, the hardware processor attempts to execute the instruction using the emulator.

3. The system of claim 1, wherein, for each instruction of the plurality of instructions beginning at the address, the hardware processor also determines if the instruction has an invalid execution address.

4. The system of claim 1, wherein, for each instruction of the plurality of instructions beginning at the address, the hardware processor also determines if the instruction is a privileged instruction.

5. The system of claim 1, wherein the hardware processor determines if an indirect control transfer instruction that is controlled by an unchanged piece of the data is effected.

6. The system of claim 1, wherein the hardware processor determines if a return instruction that is controlled by an unchanged piece of the data is effected.

7. The system of claim 1, wherein the instruction count is a count of the number of instructions in a gadget.

8. The system of claim 1, wherein the instruction count is a count of the total number of instructions executed.

9. A method for detecting the presence of a return-oriented programming (ROP) payload in data, comprising:

identifying a potential gadget address space using a hardware processor;

initializing a virtual address space of an emulator with instructions from the potential gadget address space;

determining if a piece of the data corresponds to an address of the potential gadget address space using the hardware processor; and

in response to determining that the piece of the data corresponds to an address of the potential gadget address space;

for each instruction of a plurality of instructions beginning at the address, using the hardware processor to;

determine whether the instruction is valid;

count the instruction as part of an instruction count; and

determine whether the instruction count meets at least one threshold;

in response to determining that one of the plurality of instructions is valid and determining that the instruction count meets the at least one threshold, using the hardware processor to increase a gadget count; and

indicating, using the hardware processor, that an ROP payload is present in the data in response to the gadget count meeting a threshold greater than one.

10. The method of claim 9, wherein determining whether the instruction is valid comprises attempting to execute the instruction using the emulator.

11. The method of claim 9, further comprising, for each instruction of the plurality of instructions beginning at the address, determining, using the hardware processor, if the instruction has an invalid execution address.

12. The method of claim 9, further comprising, for each instruction of the plurality of instructions beginning at the address, determining, using the hardware processor, if the instruction is a privileged instruction.

13. The method of claim 9, further comprising determining if an indirect control transfer instruction that is controlled by an unchanged piece of the data is effected.

14. The method of claim 9, further comprising determining if a return instruction that is controlled by an unchanged piece of the data is effected.

15. The method of claim 9, wherein the instruction count is a count of the number of instructions in a gadget.

16. The method of claim 9, wherein the instruction count is a count of the total number of instructions executed.

17. A non-transitory computer-readable medium containing computer-executable instructions that, when executed by a processor, cause the processor to perform a method for detecting the presence of a return-oriented programming (ROP) payload in data, the method comprising:

identifying a potential gadget address space;

initializing a virtual address space of an emulator with instructions from the potential gadget address space;

determining if a piece of the data corresponds to an address of the potential gadget address space; and

in response to determining that the piece of the data corresponds to an address of the potential gadget address space;

for each instruction of a plurality of instructions beginning at the address;

determining whether the instruction is valid;

counting the instruction as part of an instruction count; and

determining whether the instruction count meets at least one threshold;

in response to determining that one of the plurality of instructions is valid and determining that the instruction count meets the at least one threshold, increasing a gadget count; and

indicating that an ROP payload is present in the data in response to the gadget count meeting a threshold greater than one.

18. The non-transitory computer-readable medium of claim 17, wherein determining whether the instruction is valid comprises attempting to execute the instruction using the emulator.

19. The non-transitory computer-readable medium of claim 17, wherein the method further comprises, for each instruction of the plurality of instructions beginning at the address, determining if the instruction has an invalid execution address.

20. The non-transitory computer-readable medium of claim 17, wherein the method further comprises, for each instruction of the plurality of instructions beginning at the address, determining if the instruction is a privileged instruction.

21. The non-transitory computer-readable medium of claim 17, wherein the method further comprises determining if an indirect control transfer instruction that is controlled by an unchanged piece of the data is effected.

22. The non-transitory computer-readable medium of claim 17, wherein the method further comprises determining if a return instruction that is controlled by an unchanged piece of the data is effected.

23. The non-transitory computer-readable medium of claim 17, wherein the instruction count is a count of the number of instructions in a gadget.

24. The non-transitory computer-readable medium of claim 17, wherein the instruction count is a count of the total number of instructions executed.