System and methods for advanced malware detection through placement of transition events

• US 10,169,585 B1
• Filed: 06/22/2016
• Issued: 01/01/2019
• Est. Priority Date: 06/22/2016
• Status: Active Grant

• Current Assignee: Fireeye Security Holdings US LLC
• Original Assignee: FireEye, Inc.
• Inventors: Pilipenko, Alex, Ha, Phung-Te
• Primary Examiner(s): Cribbs, Malcolm
• Application Number:
  US15/189,993
• Time in Patent Office: 923 Days
• Field of Search: None
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities

Abstract

A non-transitory storage medium including instructions that are executable by one or more processors to perform operations including instrumenting a VM is shown. The VM is used to process an object to determine whether the object is associated with malware. Logic within the VM analyzes memory allocated for a process within the VM for a point of interest (POI), the POI being an address of one of a set predetermined instructions likely to be associated with malware. The VMM detects a memory violation during processing of the object and responsive to detecting the memory violation, injects a transition event at the POI on the page on which the POI is located in memory. Further, responsive to detecting an attempted execution of the transition event, the VMM (i) emulates an instruction located at the POI, and (ii) the logic within the VM performs one or more malware detection routines.

Claims

1. A non-transitory storage medium having stored thereon instructions, the instructions being executable by one or more processors to perform operations including:

instantiating a virtual machine (VM) with a virtual machine monitor (VMM), the VM to process an object to determine whether the object is associated with malware;

performing a first analysis of memory allocated for the VM for a point of interest (POI), the point of interest being an address an instruction of a set of instructions likely to be associated with malware, the set of instructions including one or more instructions;

detecting a memory violation during processing of the object, the memory violation being an attempt to access a page in the memory allocated for the VM having a permission other than “

execute”

from which a process running within the VM is attempting to execute;

responsive to detecting the memory violation, injecting a transition event at the point of interest on the page and setting the permission of the page to “

execute only”

; and

responsive to continuing the processing of the object and detecting an attempted execution of the transition event, (i) emulating the instruction of the set of instructions corresponding to the point of interest, and (ii) performing one or more malware detection routines.

2. The storage medium of claim 1, wherein the transition event is one of an interrupt, a hardware event, or a privileged instruction, and the execution of the transition event transfers control from the VM to the VMM.

3. The storage medium of claim 1, wherein the first analysis includes determining the POI based on a known structure of an operating system running in the VM and one or more known offsets of each of the set of instructions likely to be associated with malware.

4. The storage medium of claim 1, wherein the memory allocated for the VM is analyzed by a guest kernel module of the VM.

5. The storage medium of claim 1, wherein the memory violation is detected by a memory violation handler of the VMM.

6. The storage medium of claim 1, wherein the transition event is injected by a virtual machine monitor introspection logic (VMMI) of the VMM.

7. The storage medium of claim 1, wherein the point of interest is emulated by the VMMI.

8. The storage medium of claim 1, wherein the one or more malware detection routines are performed by the VM operating in kernel mode.

9. The storage medium of claim 1, wherein the instructions being executable by the one or more processors to perform operations further including:

responsive to completion of the one or more malware detection routines, continuing processing of the object by the VM operating in user mode.

10. The storage medium of claim 1, wherein the instructions being executable by the one or more processors to perform operations further including:

populating one or more data structures with each point of interest detected during the first analysis of the memory allocated for the VM.

11. The storage medium of claim 1, wherein the instructions being executable by the one or more processors to perform operations further including:

performing a second analysis of at least a portion of the memory allocated for the VM when;

(i) a new application is launched within the VM, (ii) a running application within the VM terminates, or (iii) an executable module is loaded or unloaded for an application running within the VM.

12. An electronic device comprising:

one or more processors;

a storage device communicatively coupled to the one or more processors and storing logic, the logic being executable by the one or more processors to perform operations including;

instrumenting a virtual machine (VM) with a virtual machine monitor (VMM), the virtual machine to process an object to determine whether the object is associated with malware;

performing a first analysis of memory allocated for the VM for a point of interest (POI), the point of interest being an address of an instruction of a set of instructions likely to be associated with malware, the set of instructions including one or more instructions;

detecting a memory violation during processing of the object, the memory violation being an attempt to access a page in the memory allocated for the VM having a permission other than “

execute”

from which a process running within the VM is attempting to execute;

responsive to detecting the memory violation, injecting a transition event at the point of interest on the page and setting the permission of the page to “

execute only”

; and

responsive to continuing the processing of the object and detecting an attempted execution of the transition event, (i) emulating the instruction corresponding to the point of interest, and (ii) performing one or more malware detection routines.

13. The electronic device of claim 12, wherein the transition event includes at least one of an interrupt or a privileged instruction, and the execution of the transition event transfers control from the VM to the VMM.

14. The electronic device of claim 12, wherein the first analysis includes determining the POI based on a known structure of an operating system running in the VM and one or more known offsets of the set of instructions likely to be associated with malware.

15. The electronic device of claim 12, wherein the transition event is injected by a virtual machine monitor introspection logic (VMMI) of the VMM.

16. The electronic device of claim 12, wherein the point of interest is emulated by the VMMI.

17. The electronic device of claim 12, wherein the one or more malware detection routines are performed by the VM operating in kernel mode.

18. The electronic device of claim 12, wherein the instructions being executable by the one or more processors to perform operations further including:

responsive to completion of the one or more malware detection routines, continuing processing of the object by the VM operating in user mode.

19. A method for detecting whether an object is associated with malware through processing of the object within a virtual machine (VM), the method comprising:

instrumenting the VM with a virtual machine monitor (VMM);

performing a first analysis of memory allocated for the VM for a point of interest (POI), the point of interest being an address of an instruction of a set of instructions likely to be associated with malware, the set of instructions including one or more instructions;

detecting a memory violation during processing of the object, the memory violation being an attempt to access a page in the memory allocated for a process within the VM having a permission other than “

execute”

from which a process running within the VM is attempting to execute;

responsive to detecting the memory violation, injecting an transition event at the point of interest on the page and setting the permission of the page to “

execute only”

; and

responsive to continuing the processing of the object and detecting an attempted execution of the transition event, (i) emulating the instruction corresponding to the point of interest, and (ii) performing one or more malware detection routines.

20. The method of claim 19, wherein the transition event includes at least one of an interrupt or a privileged instruction, and the execution of the transition event transfers control from the VM to the VMM.

21. The method of claim 19, wherein the first analysis includes determining the POI based on a known structure of an operating system running in the VM and one or more known offsets of the set of instructions likely to be associated with malware.

22. The method of claim 19, wherein the transition event is injected by a virtual machine monitor introspection logic (VMMI) of the VMM.

23. The method of claim 19, wherein the point of interest is emulated by the VMMI.

24. The method of claim 19, wherein the one or more malware detection routines are performed by the VM operating in kernel mode.

25. The electronic device of claim 19, wherein the instructions being executable by the one or more processors to perform operations further including:

responsive to completion of the one or more malware detection routines, continuing processing of the object by the VM operating in user mode.

26. A method for detecting whether an object is associated with malware through processing of the object within a virtual machine (VM), the method comprising:

performing a first analysis of memory allocated for the VM for one or more first points of interest (POIs), each point of interest being an address of an instruction of a set of instructions likely to be associated with malware, the set of instructions including one or more instructions;

recording each of the one or more points of interest in one or more data structures in memory shared between the VM and a virtual machine monitor (VMM); and

performing a second analysis of at least a portion of the memory allocated for a process within the VM for one or more second points of interest, the second analysis being performed in response to any of the following;

(i) a new application starts within the VM, (ii) a running application within the VM terminates, or (iii) an executable module is loaded or unloaded for a running application within the VM, and each of the one or more second points of interest are recorded in the one or more data structures;

injecting, by a virtual machine monitor introspection logic (VMMI), a transition event into a page of the memory allocated for the VM upon referencing the one or more data structures and setting a permission of the page to “

execute only”

in response to detection of a memory violation during processing of the object; and

responsive to the processing the object and detecting an attempted execution of the transition event, (i) emulating the instruction corresponding to a point of interest of either the first points of interest or the second points of interest, and (ii) performing one or more malware detection routines.

27. The method of claim 26 further comprising:

instrumenting the VM with a software profile, the software profile including one or more applications, wherein the first analysis includes analyzes memory for each running application within the VM.

28. The method of claim 26, wherein the first analysis and the second analysis are performed by a guest kernel module located within the VM.

29. The method of claim 26 wherein the transition event includes at least one of an interrupt or a privileged instruction, and the execution of the transition event transfers control from the VM to the VMM.