Securing customer virtual machines in a multi-tenant cloud

• US 8,909,928 B2
• Filed: 03/10/2011
• Issued: 12/09/2014
• Est. Priority Date: 06/02/2010
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Ahmad, Irfan, Chaturvedi, Abhishek, Kiriansky, Vladimir, Gunti, Mukund
• Primary Examiner(s): KOSOWSKI, CAROLYN M
• Application Number:
  US13/045,212
• Publication Number:
  US 20110302415A1
• Time in Patent Office: 1,370 Days
• Field of Search: 713/168
• US Class Current: 713/168
• CPC Class Codes: G06F 21/51 at application loading time, e.g. accepting, rejecting, starting or inhibiting executable software based on integrity or source reliability
• CPC Class Codes: G06F 21/57 Certifying or maintaining trusted computer platforms, e.g. secure boots or power-downs, version controls, system software checks, secure updates or assessing vulnerabilities
• CPC Class Codes: G06F 21/575 Secure boot
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 9/08 Key distribution or management, e.g. generation, sharing or updating, of cryptographic keys or passwords network architectures or network communication protocols for supporting key management in a packet data network H04L63/06
• CPC Class Codes: H04L 9/0825 using asymmetric-key encryption or public key infrastructure [PKI], e.g. key signature or public key certificates
• CPC Class Codes: H04L 9/083 involving central third party, e.g. key distribution center [KDC] or trusted third party [TTP]
• CPC Class Codes: H04L 9/0897 involving additional devices, e.g. trusted platform module [TPM], smartcard or USB

Abstract

A trusted virtualization platform protects sensitive customer data during operation of virtual machines in a multi-tenant cloud computing center. The trusted virtualization platform limits administrator access to the data and state of the virtual machines running thereon, reports any changes made thereto, and requires keys provided by the customer or a trusted third party of the customer to perform management operations on the virtual machines. By requiring cloud computing centers to use such trusted virtualization platforms, customers uploading their virtual machines into the cloud computing center can be assured that cloud administrators will not be able to access or tamper with their private data. Furthermore, customers can directly audit all important state or configuration changes for their virtual machines as the trusted virtualization platform can be configured to report all such changes according to a security policy set by the customer.

Claims

1. A method of securing virtual machines in a multi-tenant data center including a plurality of server computers and persistent data stores, comprising:

configuring a server computer with an attestation module;

installing a software stack on the server computer;

measuring, with the attestation module, a static property of the software stack and storing the measurement in the attestation module;

receiving a request to start a virtual machine using the software stack on the server computer;

in response to receiving the request to start the virtual machine, transmitting the measurement stored in the attestation module to an external entity and, in response to a verification that the measurement matches an expected value at the external entity, receiving from the external entity a key for running the virtual machine using the software stack;

running the virtual machine on top of the software stack using the key;

receiving a request to execute a management operation on the virtual machine; and

creating an audit trail by sending the management operation to the external entity.

2. The method of claim 1, wherein the external entity is an owner of the virtual machine or a third party to whom the owner has delegated security monitoring responsibilities.

3. The method of claim 1, wherein the software stack does not permit selected management operations to be performed on the virtual machine without permission from the external entity.

4. The method of claim 3, further comprising:

determining that the requested management operation is one of the selected management operations; and

based on the determination that the requested management operation is one of the selected management operations, creating the audit trail by sending the management operation to the external entity;

wherein the selected management operations include migration of the virtual machine to another server computer, creation of a snapshot of the virtual machine, and in-place upgrade of the virtual machine.

5. The method of claim 1, wherein the software stack prevents the virtual machine from communicating with an identified virtual machine running on another sever computer without an encryption key.

6. The method of claim 5, further comprising:

transmitting a request for the encryption key to the external entity, the request including an identification of the virtual machine running on another sever computer.

7. The method of claim 1, wherein the software stack has multiple layers and a static property measurement is carried out incrementally each time a software layer is added.

8. The method of claim 7, wherein the attestation module is a trusted platform module (TPM) having multiple registers and each incremental static property measurement is recorded in one of the registers of the TPM.

9. The method of claim 1, further comprising:

determining that the management operation is a restricted management operation; and

sending a notification of the restricted management operation to the external entity before the restricted management operation is carried out and after the restricted management operation is carried out.

10. The method of claim 1, further comprising:

determining that the management operation is a request to debug a certain class of issues;

based on the determination that the management operation is a request to debug a certain class of issues, creating a shell mode of the virtual machine on the software stack, the shell mode providing shell access to the server computer and hypervisor running thereon.

11. The method of claim 10, wherein the shell mode is enabled to be created only after the software stack scrubs all memory and storage that can potentially leak user data.

12. The method of claim 11, further comprising:

after the scrubbing, extending a hash of the shell'"'"'s binary code into a Platform Configuration Registers (PCR) where the hash has been recorded to reflect that a shell has been launched on the server computer; and

prohibiting the virtual machine from being restarted on the server computer without going through a reboot process.

13. A non-transitory machine readable storage medium for securing virtual machines in a multi-tenant data center including a plurality of server computers and persistent data stores, the machine readable storage medium having computer instructions encoded thereon causing a computer configured as a trusted virtualization platform to perform a method, the method comprising:

receiving a request for a key to run a virtual machine on a server computer configured with an attestation module, the key request initiated by a request for a virtual machine received at the server computer, the key request including a customer ID associated with the virtual machine;

in response to the key request, requesting the server computer for static property measurements of a software stack on top which the virtual machine will be run, the static property measurements stored in the attestation module on the server computer;

receiving the static property measurements stored in the attestation module from the server computer;

confirming from the static property measurements that the software stack is a trusted software stack by verifying that the static property measurements stored in the attestation module match an expected value and, after said confirming, transmitting to the server computer the key to run the virtual machine on the server computer;

receiving a request to execute a management operation on the virtual machine; and

creating an audit trail by storing the requested management operation.

20. The machine readable storage medium of claim 13, wherein the method further comprises:

determining that the requested management operation is a security breach;

request a remediation action based on the security breach.

21. The machine readable storage medium of claim 20, wherein the remedial action is one or more of the following:

a request to power off the virtual machine;

sending an alert to the customer; and

migrating the virtual machine to a private cloud associated with the customer.

14. The machine readable storage medium of 13, wherein the method further comprises:

receiving a public key of the attestation module from the server computer; and

searching for the public key in an inventory associated with the multi-tenant data center,

wherein the key to run the virtual machine on the server computer is transmitted to the server computer after it is confirmed that the public key has been found in the inventory of associated with the multi-tenant data center.

15. The machine readable storage medium of 13, wherein the method further comprises:

receiving a request to perform an operation on the virtual machine running on the server computer; and

examining a policy associated with the virtual machine and transmitting a key for performing the operation on the virtual machine if the policy permits the operation and transmitting a message denying the request if the policy does not permit the operation.

16. The machine readable storage medium of 13, wherein the method further comprises:

after the virtual machine is running on the server computer, transmitting a key for encrypting and decrypting data stored in a virtual disk of the virtual machine.

17. The machine readable storage medium of 13, wherein the method further comprises:

receiving a request to transmit a network packet to another virtual machine running in a different server computer;

confirming that the different server computer is running a trusted software stack and said another virtual machine is running on top of the trusted software stack; and

upon said confirming, transmitting to the server computer a key for encrypting the network packet to be transmitted to said another virtual machine.

18. The machine readable storage medium of 13, wherein the method further comprises:

receiving a public portion of an attestation identification key (AIK) from the server computer,

wherein the static property measurements are encrypted with a private portion of the AIK and decrypted using the public portion of the AIK.

19. The machine readable storage medium of 13, wherein the method further comprises:

transmitting a random nonce to the server computer; and

confirming that the static property measurements were actually transmitted by the server computer when the random nonce is also received from the server computer.

22. A multi-tenant data center system comprising:

a plurality of server computers, each of which is configured with a trusted platform module (TPM) and a trusted virtualization platform having one or more software layers on top of which a virtual machine is to be executed; and

a persistent storage system coupled to the server computers, in which files for launching the virtual machine are stored, the files including an encrypted portion and a plain text portion that identifies a customer and a network location associated with the customer,

wherein the trusted virtualization platform is programmed to;

create an audit trail by reporting any changes to the trusted virtualization platform to the network location associated with the customer;

in response to receiving a request for a virtual machine, obtain one or more keys from the network location associated with the customer to decrypt the encrypted portion of the files for launching the customer application, the one or more keys obtained by transmitting one or more measurements stored in the TPM to the network location and receiving the one or more keys if the one or more measurements match expected values in the network location;

receiving a request to execute a management operation on the virtual machine; and

adding the requested management operation to the audit trail by sending the management operation to the network location associated with the customer.

23. The system of claim 22, wherein the one or more keys are stored in the memory of the virtual machine and are inaccessible by an administrator.