System and method for protecting a software component running in a virtual machine through virtual interrupts by the virtualization layer

• US 10,726,127 B1
• Filed: 06/30/2016
• Issued: 07/28/2020
• Est. Priority Date: 06/30/2015
• Status: Active Grant

• Current Assignee: Musarubra US LLC (Musarubra US SellCo LLC)
• Original Assignee: FireEye, Inc. (Alphabet Inc.)
• Inventors: Steinberg, Udo
• Primary Examiner(s): Callahan, Paul E
• Application Number:
  US15/199,882
• Time in Patent Office: 1,489 Days
• CPC Class Codes: G06F 2009/45587 Isolation or security of virtual machine instances
• CPC Class Codes: G06F 21/53 by executing in a restricted environment, e.g. sandbox or secure virtual machine
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
• CPC Class Codes: G06F 21/629 to features or functions of an application
• CPC Class Codes: G06F 2221/034 Test or assess a computer or a system
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: G06F 9/4812 by interrupt, e.g. masked

Abstract

A computing device features one or more hardware processors and a memory that is coupled to the one or more processors. The memory comprises software that is implemented with a security mechanism to protect the availability of a software component operating within a virtual machine, which is controlled by a guest operating system (OS) kernel. The software comprises a virtualization layer operating in a host mode, where the virtualization layer, when executed by the one or more hardware processors, is configured to send one or more virtual interrupts to the guest OS kernel of the virtual machine. A virtual interrupt causes an interrupt service routine within the guest OS kernel to perform a particular service that prevents a protected process (or protected software data structures) from being effected by malware.

Claims

1. A computing device comprising:

one or more hardware processors; and

a memory coupled to the one or more processors, the memory comprises software that supports virtualization including (i) a virtual machine operating in a guest mode and controlled by a guest operating system (OS) kernel and (ii) a virtualization layer operating in a host mode,

wherein the virtualization layer being configured to send one or more virtual interrupts to the guest OS kernel of the virtual machine that causes an interrupt service routine within the guest OS kernel to perform a particular service to determine whether a protected process running in the virtual machine is active and, responsive to determining that the protected process is inactive, alter one or more permissions for a memory page associated with the protected process by removing the one or more permission from at least one nested page table entry that is associated with the memory page containing code pertaining to the protected process.

2. The computing device of claim 1, wherein the virtualization layer, when executed by the one or more hardware processors, is configured to send the one or more virtual interrupts that cause the interrupt service routine to perform the particular service that prompts the protected process to restart if inactive.

3. The computing device of claim 1, wherein the virtualization layer, when executed by the one or more hardware processors, is configured to periodically send the one or more virtual interrupts to the interrupt service routine within the guest OS kernel, the interrupt service routine to perform the particular service that includes either (i) checking an integrity of a portion of the guest OS, or (ii) checking integrity of a data structure of the protected process, or (iii) requesting a response message from the protected process to verify that the protected process is not disabled.

4. The computing device of claim 1, wherein the virtualization layer includes a guest monitor component that, when executed by the one or more hardware processors, sends the one or more virtual interrupts to the interrupt service routine at a predetermined rate.

5. The computing device of claim 4, wherein the one or more virtual interrupts to cause execution of the guest OS kernel to vector to the interrupt service routine at a next instruction boundary.

6. The computing device of claim 1, wherein the virtualization layer, when executed by the one or more hardware processors, is configured to periodically send the one or more virtual interrupts to the interrupt service routine within the guest OS kernel in response to detection of a particular event and the one or more virtual interrupts causing the protected process to remain in an active state.

7. The computing device of claim 6, wherein the particular event includes an unpermitted operation attempted on a memory page associated with a nested page table that is referenced for a memory address translation from a guest physical address to a host physical address.

8. The computing device of claim 1, wherein the interrupt service routine within the guest OS kernel preventing the protected process running in the virtual machine from being controlled by malware comprises ensuring that the protected process, providing enhanced security to the computing device, is operating in the virtual machine as confirmed by the protected process assisting in servicing of the one or more virtual interrupts.

9. The computing device of claim 1, wherein the particular service includes checking whether the protected process running in the virtual machine is in operation, the protected process monitoring behaviors of an object being analyzed for malware during run time of the object within the virtual machine and capturing any resulting events that occur during the run-time.

10. The computing device of claim 1, wherein the particular service includes restarting the protected process running in the virtual machine if the protected process is disabled, the protected process monitoring behaviors of an object being analyzed for malware during run time of the object within the virtual machine and capturing any resulting events that occur during the run-time.

11. A computerized method for protecting a protected process comprising:

sending one or more virtual interrupts by a virtualization layer operating in a host mode to a guest operating system (OS) kernel, the guest OS kernel to control a virtual machine operating in a guest mode;

receiving the one or more virtual interrupts by an interrupt service routine within the guest OS kernel; and

responsive to receiving the one or more virtual interrupts, performing a particular service by the interrupt service routine to determine whether a protected process running in the virtual machine is active and removing one or more permissions controlling access to a memory page associated with the protected process upon determining that the protected process is inactive so as to prevent the protected process from being controlled by malware,

wherein the protected process includes a guest agent monitoring behaviors of an object being analyzed for malware during run time of the object within the virtual machine and capturing any resulting events that occur during the run-time.

12. A computerized method of claim 11, for protecting a protected process comprising:

sending one or more virtual interrupts to a guest operating system (OS) kernel, the guest OS kernel to control a virtual machine operating in a guest mode;

receiving the one or more virtual interrupts by an interrupt service routine within the guest OS kernel; and

responsive to receiving the one or more virtual interrupts, performing a particular service by the interrupt service routine to determine whether a protected process running in the virtual machine is active and removing one or more permissions controlling access to a memory page associated with the protected process upon determining that the protected process is inactive so as to prevent the protected process from being controlled by malware,

wherein the performing of the particular service comprises restarting the protected process running in the virtual machine if the protected process is disabled, the protected process is a process for monitoring behaviors of an object being analyzed for malware during run time of the object within the virtual machine and capturing any resulting events that occur during the run-time.

13. A computerized method for protecting a protected process comprising:

sending one or more virtual interrupts by a virtualization layer operating in a host mode to a guest operating system (OS) kernel, the guest OS kernel to control a virtual machine operating in a guest mode;

receiving the one or more virtual interrupts by an interrupt service routine within the guest OS kernel; and

responsive to receiving the one or more virtual interrupts, performing a particular service by the interrupt service routine to determine whether a protected process running in the virtual machine is active and removing one or more permissions controlling access to a memory page associated with the protected process upon determining that the protected process is inactive so as to prevent the protected process from being controlled by malware,

wherein the sending of the one or more virtual interrupts by the virtualization layer comprises sending the one or more virtual interrupts that cause the interrupt service routine to perform the particular service that prevents the protected process from being disabled, the interrupt service routine to perform the particular service that includes either (i) checking an integrity of a portion of the guest OS, or (ii) checking integrity of a data structure of the protected process, or (iii) requesting a response message from the protected process to verify that the protected process is not disabled.

14. The computerized method of claim 13, wherein the sending of the one or more virtual interrupts by the virtualization layer is conducted periodically.

15. The computerized method of claim 13, wherein the sending of the one or more virtual interrupts by the virtualization layer is conducted at a predetermined rate.

16. The computerized method of claim 15, wherein the one or more virtual interrupts to cause execution of the guest OS kernel to vector to the interrupt service routine at a next instruction boundary.

17. A computerized method for protecting a protected process comprising:

sending one or more virtual interrupts to a guest operating system (OS) kernel, the guest OS kernel to control a virtual machine operating in a guest mode;

receiving the one or more virtual interrupts by an interrupt service routine within the guest OS kernel; and

responsive to receiving the one or more virtual interrupts, performing a particular service by the interrupt service routine to determine whether a protected process running in the virtual machine is active and removing one or more permissions controlling access to a memory page associated with the protected process upon determining that the protected process is inactive so as to prevent the protected process from being controlled by malware,

wherein the sending of the one or more virtual interrupts to the interrupt service routine is conducted periodically in response to detection of a particular event and the one or more virtual interrupts causing the protected process to remain in an active state.

18. The computerized method of claim 17, wherein the particular event includes an unpermitted operation attempted on a memory page associated with a nested page table that is referenced for a memory address translation from a guest physical address to a host physical address.

19. A computerized method for protecting a protected process comprising:

sending one or more virtual interrupts by a virtualization layer operating in a host mode to a guest operating system (OS) kernel, the guest OS kernel to control a virtual machine operating in a guest mode;

receiving the one or more virtual interrupts by an interrupt service routine within the guest OS kernel; and

responsive to receiving the one or more virtual interrupts, performing a particular service by the interrupt service routine to determine whether a protected process running in the virtual machine is active and removing one or more permissions controlling access to a memory page associated with the protected process upon determining that the protected process is inactive so as to prevent the protected process from being controlled by malware,

wherein the interrupt service routine within the guest OS kernel preventing the protected process running in the virtual machine from being controlled by malware by ensuring that the protected process, providing enhanced security to the computing device, is operating in the virtual machine as confirmed by the protected process assisting in servicing the one or more virtual interrupts.

20. A non-transitory storage medium including software that, upon execution by one or more processors, performs operations to protect one or more processes being performed by a computing device from being compromised through a malicious attack, the non-transitory storage medium comprising:

a virtual machine operating in a guest mode and controlled by a guest OS kernel; and

a virtualization layer operating in a host mode, wherein the virtualization layer, during execution, is configured to send one or more virtual interrupts to the guest OS kernel of the virtual machine that causes an interrupt service routine within the guest OS kernel to (i) perform a particular service to determine whether a protected process running in the virtual machine is active and, responsive to determining that the protected process is inactive and (ii) alter one or more permissions for a memory page associated with the protected process by removing the one or more permission from at least one nested page table entry that is associated with the memory page containing code pertaining to the protected process.

21. The non-transitory storage medium of claim 20, wherein the virtualization layer is configured to send the one or more virtual interrupts that cause the interrupt service routine to perform the particular service that prompts the protected process to restart if inactive.

22. The non-transitory storage medium of claim 20, wherein the virtualization layer is configured to periodically send the one or more virtual interrupts to the interrupt service routine within the guest OS kernel, the interrupt service routine to perform the particular service that includes either (i) checking an integrity of a portion of the guest OS, or (ii) checking integrity of a data structure of the protected process, or (iii) requesting a response message from the protected process to verify that the protected process is not disabled.

23. The non-transitory storage medium of claim 20, wherein the virtualization layer includes a guest monitor component that sends the one or more virtual interrupts to the interrupt service routine at a predetermined rate.

24. The non-transitory storage medium of claim 23, wherein the one or more virtual interrupts sent by the virtualization layer, during execution, is intended to cause execution of the guest OS kernel to vector to the interrupt service routine at a next instruction boundary.

25. The non-transitory storage medium of claim 20, wherein the virtualization layer is configured to periodically send the one or more virtual interrupts to the interrupt service routine within the guest OS kernel in response to detection of a particular event and the one or more virtual interrupts causing the protected process to remain in an active state.

26. The non-transitory storage medium of claim 25, wherein the particular event includes an unpermitted operation attempted on a memory page associated with a nested page table that is referenced for a memory address translation from a guest physical address to a host physical address.

27. The non-transitory storage medium of claim 20, wherein the virtualization layer causes the interrupt service routine within the guest OS kernel to prevent the protected process running in the virtual machine from being controlled by malware by at least ensuring that the protected process, providing enhanced security to the computing device, is operating in the virtual machine as confirmed by the protected process assisting in servicing of the one or more virtual interrupts.

28. The non-transitory storage medium of claim 20, wherein the virtualization layer performs the particular service by at least checking whether the protected process running in the virtual machine is in operation, the protected process monitoring behaviors of an object being analyzed for malware during run time of the object within the virtual machine and capturing any resulting events that occur during the run-time.

29. The non-transitory storage medium of claim 20, wherein the virtualization layer performs the particular service by at least restarting the protected process running in the virtual machine if the protected process is disabled, the protected process monitoring behaviors of an object being analyzed for malware during run time of the object within the virtual machine and capturing any resulting events that occur during the run-time.