System and method for protecting memory pages associated with a process using a virtualization layer

• US 10,216,927 B1
• Filed: 06/30/2016
• Issued: 02/26/2019
• Est. Priority Date: 06/30/2015
• Status: Active Grant

• Current Assignee: Fireeye Security Holdings US LLC
• Original Assignee: FireEye, Inc.
• Inventors: Steinberg, Udo
• Primary Examiner(s): Thai, Tuan V
• Application Number:
  US15/199,879
• Time in Patent Office: 971 Days
• CPC Class Codes: G06F 12/1425 the protection being physical, e.g. cell, word, block
• CPC Class Codes: G06F 2009/45587 Isolation or security of virtual machine instances
• CPC Class Codes: G06F 21/53 by executing in a restricted environment, e.g. sandbox or secure virtual machine
• CPC Class Codes: G06F 2212/1052 Security improvement
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects

Abstract

A computerized method is provided for protecting processes operating within a computing device. The method comprises an operation for identifying, by a virtualization layer operating in a host mode, when a guest process switch has occurred. The guest process switch corresponds to a change as to an operating state of a process within a virtual machine. Responsive to an identified guest process switch, an operation is conducted to determine, by the virtualization layer, whether hardware circuitry within the computing device is to access a different nested page table for use in memory address translations. The different nested page table alters page permissions for one or more memory pages associated with at least the process that are executable in the virtual machine.

Claims

1. A computerized method for protecting processes operating within a computing device, comprising:

identifying, by a virtualization layer operating in a host mode and being executed by hardware circuitry, when a guest process switch has occurred, the guest process switch corresponds to a change as to an operating state of a process within a virtual machine that is detected by (i) a change in a data store associated with the hardware circuitry and (ii) the data store stores a value different from an address space associated with the guest agent process; and

responsive to the identified guest process switch, determining, by the virtualization layer, whether the hardware circuitry within the computing device is to access a different nested page table for use in memory address translations, wherein the different nested page table alters page permissions for one or more memory pages associated with one or more processes including the process that are executable in the virtual machine.

2. The computerized method of claim 1, wherein the determining whether the hardware circuitry within the computing device is to access the different nested page table for use in memory address translations comprises assigning the different nested page table that restricts the page permissions for the one or more memory pages associated with the process when the process transitions from an active operating state to an inactive operating state.

3. The computerized method of claim 1, wherein the identifying when the guest process switch has occurred comprises analyzing the change in content within the data store associated with the hardware circuitry that is executing the virtualization layer. The data store being a control register of a processor.

4. The computerized method of claim 1, wherein the determining whether the hardware circuitry within the computing device is to access the different nested page table comprises determining that the guest process switch identifies that the process is now inactive and removal of one or more page permissions for one or more memory pages associated with the process are to be performed.

5. The computerized method of claim 4, wherein one or more page permissions includes removal of a write permission for the one or more memory pages associated with the process.

6. The computerized method of claim 4, wherein one or more page permissions includes removal of a write permission, a read permission and an execute permission for the one or more memory pages associated with the process so that the one or more memory pages are inaccessible inside the virtual machine.

7. The computerized method of claim 1, wherein the determining whether the hardware circuitry within the computing device is to access the different nested page table comprises determining that the guest process switch identifies that the process is now inactive and altering one or more page permissions for one or more memory pages within a memory that handle a translation from a guest-physical address (GPA) to a host-physical address (HPA).

8. The computerized method of claim 1, wherein the process monitors and stores information associated with analyzed content or events that may be associated with malicious activity during processing of an object within the virtual machine.

9. The computerized method of claim 1, wherein the virtualization layer controls access by the hardware circuitry to the at least one different nested page table to restrict the page permissions for the one or more memory pages associated with the guest agent process when the content of the data store corresponding to an address space different than the address space associated with the guest agent process.

10. A computing device, comprising:

a hardware processor including a control register;

a memory coupled to the hardware processor, the memory includes virtualization software that includes a virtual machine to operate in a guest mode and a virtualization layer to operate in a host mode,

wherein in response to detecting a guest process switch corresponding to a change to an operating state of a guest agent process within a virtual machine as being set into an inactive operating state, the virtualization layer, being executed by the processor, assigns a page table for use in memory address translations different than a page table currently in use, the different page table alters page permissions for one or more memory pages associated with the guest agent process, and

wherein the virtualization layer detects the guest process switch by detecting a change in content with the control register of the processor that is executing the virtualization layer and the virtualization layer assigns the different page table to restrict the page permissions for the one or more memory pages associated with the guest agent process when the content of the control register corresponds to an address space different than an address space associated with the guest agent process.

11. The computing device of claim 10, wherein the virtualization layer assigns the different page table that restricts the page permissions for the one or more memory pages associated with the guest agent process when the guest agent process transitions from an active operating state to the inactive operating state.

12. The computing device of claim 11, wherein the page permissions are restricted by removing a write permission for the one or more memory pages associated with the guest agent process.

13. The computing device of claim 11, wherein the page permissions are restricted by removing an execute permission for the one or more memory pages associated with the guest agent process.

14. The computing device of claim 11, wherein the page permissions are restricted by removing a write permission, a read permission and an execute permission for the one or more memory pages associated with the guest agent process so that the one or more memory pages are inaccessible inside the virtual machine.

15. The computing device of claim 10, wherein the virtualization layer detects the guest process switch by detecting a change in content with the control register of the processor that is executing the virtualization layer.

16. The computing device of claim 10, wherein the page permissions are restricted by a guest monitor component of the virtualization layer removing a write permission for the one or more memory pages associated with the guest agent process.

17. The computing device of claim 10, wherein the page permissions are restricted by a guest monitor component of the virtualization layer removing an execute permission for the one or more memory pages associated with the guest agent process.

18. The computing device of claim 10, wherein the page permissions are restricted by a guest monitor component of the virtualization layer removing a write permission, a read permission and an execute permission for the one or more memory pages associated with the guest agent process so that the one or more memory pages are inaccessible inside the virtual machine.

19. The computing device of claim 10, wherein the page table includes a first set of nested page tables include one or more memory pages within the memory to handle a first translation from a guest-physical address to a host-physical address and the different page table includes a second set of nested page tables include one or more memory pages within the memory to handle a second translation from a guest-physical address (GPA) to a host-physical address, the second transition being different from the first transition.

20. An endpoint device, comprising:

a system interconnect;

a hardware processor coupled to the system interconnect, the hardware processor includes a control register;

a network interface coupled to the system interconnect, the network interface being configured to transmit or receive messages in accordance with a selected communication protocol; and

a memory coupled to the system interconnect, the memory includes virtualization software that includes a virtual machine to operate in a guest mode and a virtualization layer to operate in a host mode,

wherein in response to detecting a guest process switch corresponding to a change to an operating state of a guest agent process within a virtual machine as being set into an inactive state, the virtualization layer, being executed by the processor, assigns at least one nested page table for use in memory address translations different than nested page tables currently in use, the at least one different nested page table restricting page permissions for one or more memory pages associated with guest agent process while being set into the inactive state, and

wherein the virtualization layer assigns the at least one different nested page table to restrict the page permissions for the one or more memory pages associated with the guest agent process when content of the control register corresponds to an address space different than an address space associated with the guest agent process.