Method to detect forgery and exploits using last branch recording registers

• US 10,621,338 B1
• Filed: 06/29/2016
• Issued: 04/14/2020
• Est. Priority Date: 12/30/2015
• Status: Active Grant

• Current Assignee: Fireeye Security Holdings US LLC
• Original Assignee: FireEye, Inc.
• Inventors: Pfoh, Jonas, Ha, Phung-Te
• Primary Examiner(s): Pwu, Jeffrey C
• Assistant Examiner(s): Ambaye, Samuel
• Application Number:
  US15/197,656
• Time in Patent Office: 1,385 Days
• Field of Search: 726 23
• CPC Class Codes: G06F 2009/45587 Isolation or security of virtual machine instances
• CPC Class Codes: G06F 2009/45595 Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 21/54 by adding security routines or objects to programs
• CPC Class Codes: G06F 21/554 involving event detection and direct action
• CPC Class Codes: G06F 2221/033 Test or assess software
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms

Abstract

A method for detecting a ROP attack comprising processing of an object within a virtual machine managed by a virtual machine monitor (VMM), intercepting an attempted execution by the object of an instruction, the instruction stored on a page in memory that is accessed by the virtual machine, responsive to determining the page includes instructions corresponding to one of a predefined set of function calls, (i) inserting a first transition event into the memory at a starting address location of a function call, and (ii) setting a permission of the page to be execute only, and responsive to triggering the first transition event, halting, by the VMM, the processing of the object and analyzing, by logic within the VMM, content of last branch records associated with the virtual machine to determine whether the processing of the object displays characteristics of a ROP attack is shown.

Claims

1. A non-transitory storage medium having stored thereon logic, the logic being executable by one or more processors to perform operations including:

intercepting an attempted execution of an instruction resulting from processing of the object within a virtual machine, the instruction located on a page in memory that is accessed by the virtual machine;

responsive to determining the page includes instructions corresponding to a function call of a predefined set of function calls, (i) inserting a first transition event into the memory at a starting address location of the function call of the predefined set of function calls, and (ii) setting a permission of the page to be execute only; and

responsive to a triggering of the first transition event, (1) halting, by a virtual machine monitor, the processing of the object, and (2) analyzing, by logic within the virtual machine monitor, content of one or more last branch records associated with the virtual machine to determine whether the processing of the object displays characteristics of a return-oriented programming attack, wherein the analyzing includes;

(i) parsing the one or more last branch records associated with the virtual machine, and

(ii) responsive to detecting a return instruction as a branching instruction and determining an instruction immediately preceding a location branched to by the return instruction is not a call instruction type, determining the processing of the object displays characteristics of the return-oriented programming attack.

2. The non-transitory storage medium of claim 1, wherein the instructions being executable by the one or more processors to perform operations further including:

receiving the object within network traffic;

performing a static analysis on the object; and

responsive to determining the object is at least suspicious, providing the object to the virtual machine for processing.

3. The non-transitory storage medium of claim 1, wherein a permission of each page in memory associated with the virtual machine is set to prohibit execution when the processing of the object begins.

4. The non-transitory storage medium of claim 1, wherein the instructions being executable by the one or more processors to perform operations further including:

responsive to determining the processing displays characteristics of the return-oriented programming attack, generating an alert indicating the processing displays characteristics of the return-oriented programming attack.

5. The non-transitory storage medium of claim 1, wherein the first transition event includes a breakpoint.

20. The storage medium of claim 1, wherein prior to intercepting an attempted execution, the method further comprising:

processing of the object within the virtual machine monitored by the virtual machine monitor.

6. An electronic device comprising:

one or more processors;

a storage device for storing one or more instructions, the one or more instructions being executable by the one or more processors to perform operations including;

processing of an object within a virtual machine managed by a virtual machine monitor,

intercepting an initial attempted execution of an instruction resulting from the processing the object, the instruction located on a page in memory associated with the virtual machine;

responsive to determining the page includes instructions corresponding to a function call of a predefined set of function calls, (i) inserting a first transition event into the memory associated with the virtual machine at a location on the page of a first instruction of the instructions corresponding to the function call of the predefined set of function calls, and (ii) setting a permission of the page to be execute only, and

responsive to a triggering of the first transition event, (1) halting, by the virtual machine monitor, the processing of the object, and (2) analyzing, by logic within the virtual machine monitor, content of one or more last branch records associated with the virtual machine to determine whether the processing of the object displays characteristics of a return-oriented programming attack, wherein the analyzing includes;

(i) parsing the content of the one or more last branch records associated with the virtual machine, and

(ii) responsive to detecting a return instruction as a branching instruction and determining an instruction immediately preceding a location branched to by the return instruction is not a call instruction type, determining the processing of the object displays characteristics of the return-oriented programming attack.

7. The electronic device of claim 6, wherein the logic being executable by the one or more processors to perform operations further including:

receiving the object within network traffic;

performing a static analysis on the object; and

responsive to determining the object is at least suspicious, providing the object to the virtual machine for processing.

8. The electronic device of claim 6, wherein a permission of each page in memory associated with the virtual machine is set to prohibit execution when the processing of the object begins.

9. The electronic device of claim 6, wherein the logic being executable by the one or more processors to perform operations further including:

responsive to determining the processing of the object displays characteristics of the return-oriented programming attack, generating an alert indicating the processing displays characteristics of the return-oriented programming attack.

10. The electronic device of claim 6, wherein the first transition event includes a breakpoint.

11. A method for detecting a return-oriented programming attack comprising:

processing of an object within a virtual machine managed by a virtual machine monitor;

intercepting an initial attempted execution of an instruction resulting from the processing of the object, the instruction located on a page in memory associated with the virtual machine;

responsive to determining the page includes instructions corresponding to a function call of a predefined set of function calls, (i) inserting a first transition event into the memory associated with the virtual machine at a location on the page of a first instruction of the instructions corresponding to the function call of the predefined set of function calls, and (ii) setting a permission of the page to be execute only; and

responsive to a triggering of the first transition event, (1) halting, by the virtual machine monitor, the processing of the object, and (2) analyzing, by logic within the virtual machine monitor, content of one or more last branch records associated with the virtual machine to determine whether the processing of the object displays characteristics of a return-oriented programming attack, wherein the analyzing includes;

(i) parsing the one or more last branch records associated with the virtual machine, and

(ii) responsive to detecting a return instruction as a branching instruction and determining an instruction immediately preceding a location branched to by the return instruction is not a call instruction type, determining the processing of the object displays characteristics of the return-oriented programming attack.

12. The method of claim 11 further comprising:

receiving the object within network traffic;

performing a static analysis on the object; and

responsive to determining the object is at least suspicious, providing the object to the virtual machine for processing.

13. The method of claim 12, wherein a permission of each page in memory associated with the virtual machine is set to prohibit execution when the processing of the object begins.

14. The method of claim 11, wherein the first transition event includes a breakpoint.

15. A method for detecting a return-oriented programming attack comprising:

processing of an object within a virtual machine managed by a virtual machine monitor;

managing, by the virtual machine monitor, insertion of a transition event on an instruction located on a page in memory utilized by the virtual machine;

managing, by the virtual machine monitor, permissions of the page in memory according to the insertion of the transition event or a removal of the transition event in the instruction; and

responsive to a triggering of the transition event, analyzing, by logic within the virtual machine monitor, content of one or more last branch records associated with the virtual machine to determine whether the processing of the object displays characteristics of a return-oriented programming attack, wherein the analyzing includes;

(i) parsing the content of the one or more last branch records associated with the virtual machine, and

(ii) responsive to detecting a return instruction as a branching instruction and determining an instruction immediately preceding a location branched to by the return instruction is not a call instruction type, determining the processing of the object displays characteristics of the return-oriented programming attack.

16. The method of claim 15 further comprising:

receiving the object within network traffic;

performing a static analysis on the object; and

responsive to determining the object is at least suspicious based on the static analysis, providing the object to the virtual machine for processing.

17. The method of claim 15, wherein a permission of the page in memory is set to prohibit execution when the processing of the object begins.

18. The method of claim 15, wherein the transition event includes a breakpoint.

19. The method of claim 15, wherein the transition event includes an event that disrupts a flow of processing.

21. A method for detecting a return-oriented programming attack, comprising:

processing of an object within a virtual machine, the virtual machine being managed by a virtual machine monitor;

managing, by the virtual machine monitor, insertion of a transition event on a function of interest located on a page in memory utilized by the virtual machine;

managing, by the virtual machine monitor, permissions of the page in memory according to the insertion of the transition event or a removal of the transition event in the instruction; and

responsive to a triggering of the transition event, analyzing, by logic within the virtual machine monitor, recorded data associated with the virtual machine to determine whether the processing of the object displays characteristics of a return-oriented programming attack, wherein the analyzing includes;

(i) parsing the content of the one or more last branch records associated with the virtual machine, and (ii) responsive to detecting a return instruction as a branching instruction and determining an instruction immediately preceding a location branched to by the return instruction is not a call instruction type, determining the processing of the object displays characteristics of the return-oriented programming attack.

22. The method of claim 21 further comprising:

receiving an object within network traffic;

performing a static analysis on the object; and

responsive to determining the object is at least suspicious based on the static analysis, providing the object to the virtual machine for processing.

23. The method of claim 22, wherein a permission of the page in memory is set to prohibit execution when the processing of the object begins.

24. The method of claim 22, wherein the transition event includes a breakpoint.

25. The method of claim 21, wherein the transition event includes an event that disrupts a flow of processing.

26. The method of claim 21, wherein the recorded data associated with the virtual machine includes recorded data associated with the virtual machine, the recorded data being used to determine whether the processing of the object displays characteristics of the return-oriented programming attack.