Methods and system for hiding transition events for malware detection

• US 10,341,365 B1
• Filed: 06/30/2016
• Issued: 07/02/2019
• Est. Priority Date: 12/30/2015
• Status: Active Grant

• Current Assignee: Fireeye Security Holdings US LLC
• Original Assignee: FireEye, Inc.
• Inventors: Ha, Phung-Te
• Primary Examiner(s): Homayounmehr, Farid
• Assistant Examiner(s): Bucknor, Olanrewaju J.
• Application Number:
  US15/199,812
• Time in Patent Office: 1,097 Days
• Field of Search: None
• CPC Class Codes: G06F 2009/45587 Isolation or security of virtual machine instances
• CPC Class Codes: G06F 2009/45595 Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 21/54 by adding security routines or objects to programs
• CPC Class Codes: G06F 21/554 involving event detection and direct action
• CPC Class Codes: G06F 2221/033 Test or assess software
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms

Abstract

A method for hiding transition events during malware detection comprising processing of an object within a VM, intercepting an attempted execution of an instruction located on a page in memory associated with the VM, responsive to determining the page includes instructions corresponding to one of a predefined set of function calls, (i) inserting a first transition event into the memory at a location on the page of a first instruction of the instructions, and (ii) setting a permission of the page to be execute only, and responsive to further processing within the VM causing an attempt to read from or write to the page including the first transition event, (i) halting processing within the VM, (ii) removing the first transition event, (iii) setting the permission of the page to prohibit execution, and (iv) resuming the processing is shown.

Claims

1. A non-transitory storage medium having stored thereon logic, the logic being executable by one or more processors to perform operations including:

processing of an object within a virtual machine;

intercepting an attempted execution of an instruction by the object, the instruction located on a page in memory associated with the virtual machine;

responsive to determining the page includes instructions corresponding to one of a set of function calls, (i) inserting a first transition event into the memory associated with the virtual machine at a location of a first instruction of the instructions corresponding to a function call of the set of function calls wherein the location is on the page in the memory, and (ii) setting a permission of the page to be execute only; and

responsive to further processing within the virtual machine causing an attempt to read from or write to the page including the first transition event, (i) halting at least a portion of the processing within the virtual machine, (ii) performing an analysis of at least one last branch record (LBR) of a virtual central processing unit (CPU) of the virtual machine, and (iii) based on the analysis of the at least one LBR, determining whether the processing displays characteristics of a return-oriented programming (ROP) attack.

2. The storage medium of claim 1, wherein the transition event includes one of a function call, system call, or specified instructions, the execution of which results in halting of the processing within the virtual machine.

3. The storage medium of claim 1, wherein the processing of the object within the virtual machine is managed by a virtual machine monitor.

4. The storage medium of claim 3, wherein inserting and removing of the first transition event is performed by the virtual machine monitor.

5. The storage medium of claim 3, wherein setting of the permission is performed by the virtual machine monitor.

6. The storage medium of claim 1, wherein the instructions being executable by the one or more processors to perform operations further including:

receiving the object within network traffic;

performing a static analysis on the object; and

responsive to determining the object is at least suspicious, providing the object to the virtual machine for processing.

7. The storage medium of claim 1, wherein a permission of each page in memory associated with the virtual machine is set to prohibit execution when the processing of the object begins.

8. The storage medium of claim 1, wherein the processing of the object within the virtual machine is part of a malware detection analysis and the setting of the permission of the page to be execute only prevents malware from detecting the malware detection analysis is being conducted.

9. An electronic device comprising:

one or more processors;

a storage device including a non-transitory computer-readable medium for storing logic, the logic being executable by the one or more processors to perform operations including;

processing of an object within a virtual machine;

intercepting an attempted execution of an instruction by the object, the instruction located on a page in memory associated with the virtual machine;

responsive to determining the page includes instructions corresponding to one of a set of function calls, (i) inserting a first transition event into the memory associated with the virtual machine at a location of a first instruction of the instructions corresponding to a function call of the set of function calls, wherein the location is on the page in the memory, and (ii) setting a permission of the page to be execute only; and

responsive to further processing within the virtual machine causing an attempt to read from or write to the page including the first transition event, (i) halting at least a portion of the processing within the virtual machine, (ii) performing an analysis of at least one last branch record (LBR) of a virtual central processing unit (CPU) of the virtual machine, and (iii) based on the analysis of the at least one LBR, determining whether the processing displays characteristics of a return-oriented programming (ROP) attack.

10. The electronic device of claim 9, wherein the transition event includes one of a function call, system call, or specified instructions, the execution of which results in halting of the processing within the virtual machine.

11. The electronic device of claim 9, wherein the processing of the object within the virtual machine is managed by a virtual machine monitor.

12. The electronic device of claim 11, wherein inserting and removing of the first transition event is performed by the virtual machine monitor.

13. The electronic device of claim 11, wherein setting of the permission is performed by the virtual machine monitor.

14. The electronic device of claim 9, wherein the logic being executable by the one or more processors to perform operations further including:

receiving the object within network traffic;

performing a static analysis on the object; and

responsive to determining the object is at least suspicious, providing the object to the virtual machine for processing.

15. The electronic device of claim 9, wherein a permission of each page in memory associated with the virtual machine is set to prohibit execution when the processing of the object begins.

16. The electronic device of claim 9, wherein the processing of the object within the virtual machine is part of a malware detection analysis and the setting of the permission of the page to be execute only prevents malware from detecting the malware detection analysis is being conducted.

17. A method for hiding transition events during malware detection comprising:

processing of an object within a virtual machine;

intercepting an attempted execution of an instruction by the object, the instruction located on a page in memory associated with the virtual machine;

responsive to determining the page includes instructions corresponding to one of a set of function calls, (i) inserting a first transition event into the memory associated with the virtual machine at a location of a first instruction of the instructions corresponding to a function call of the set of function calls, wherein the location is on the page in the memory, and (ii) setting a permission of the page to be execute only; and

responsive to further processing within the virtual machine causing an attempt to read from or write to the page including the first transition event, (i) halting at least a portion of the processing within the virtual machine, (ii) performing an analysis of at least one last branch record (LBR) of a virtual central processing unit (CPU) of the virtual machine, and (iii) based on the analysis of the at least one LBR, determining whether the processing displays characteristics of a return-oriented programming (ROP) attack.

18. The method of claim 17, wherein the transition event includes one of a function call, system call, or specified instructions, the execution of which results in halting of the processing within the virtual machine.

19. The method of claim 17, wherein the processing of the object within the virtual machine is managed by a virtual machine monitor.

20. The method of claim 19, wherein inserting and removing of the first transition event is performed by the virtual machine monitor.

21. The method of claim 19, wherein setting of the permission is performed by the virtual machine monitor.

22. The method of claim 17 further comprising:

receiving the object within network traffic;

performing a static analysis on the object; and

responsive to determining the object is at least suspicious, providing the object to the virtual machine for processing.

23. The method of claim 17, wherein a permission of each page in memory associated with the virtual machine is set to prohibit execution when the processing of the object begins.

24. The method of claim 17, wherein the setting of the permission of the page to be execute only prevents malware from detecting the malware detection analysis is being conducted.

25. A method for hiding transition events during malware detection comprising:

processing an object within a virtual machine managed by a virtual machine monitor;

managing, by the virtual machine monitor, insertion of a transition event at a location of an instruction located on a page in memory utilized by the virtual machine;

managing, by the virtual machine monitor, permissions of the page in memory according to the insertion of a breakpoint or a removal of the breakpoint in the instruction;

responsive to further processing within the virtual machine causing an attempt to read from or write to the page including the transition event, (i) halting at least a portion of the processing within the virtual machine, (ii) responsive to further processing within the virtual machine causing an attempt to read from or write to the page including the first transition event, (i) halting at least a portion of the processing within the virtual machine, (ii) performing an analysis of at least one last branch record (LBR) of a virtual central processing unit (CPU) of the virtual machine, and (iii) based on the analysis of the at least one LBR, determining whether the processing displays characteristics of a return-oriented programming (ROP) attack.