On-line replacement and changing of virtualization software

• US 7,814,495 B1
• Filed: 03/31/2006
• Issued: 10/12/2010
• Est. Priority Date: 03/31/2006
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Govil, Kinshuk, Lim, Beng-Hong
• Primary Examiner(s): An; Meng-Ai
• Assistant Examiner(s): Kawsar; Abdullah Al
• Application Number:
  US11/394,558
• Time in Patent Office: 1,656 Days
• Field of Search: None
• US Class Current: 718/104
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 8/656 while running
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: G06F 9/5077 Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455

Abstract

In a virtualized system running one or more virtual machines on a first hypervisor, a second hypervisor is installed and control of the hardware resources of the physical computer supporting the virtualized system is migrated from the first hypervisor to the second hypervisor without interrupting the operation of the first hypervisor and the virtual machines. Initially a minimal set of hardware resources is hot-removed from control by the first hypervisor, and the second hypervisor is launched on the minimal set of hardware resources. Both the remaining hardware resources and the virtual machines are then migrated from the first hypervisor to the second hypervisor until all the virtual machines have been migrated over to the second hypervisor, while the virtual machines and the first hypervisor continue running largely unaffected by the migration process.

Claims

1. In a virtualized computer system running one or more virtual machines on a first hypervisor virtualizing hardware resources of a physical computer, the hardware resources including a single-path input/output (I/O) device with a single I/O path, a computer implemented method of installing a second hypervisor on the physical computer and migrating control of the hardware resources from the first hypervisor to the second hypervisor, the method comprising:

removing a predetermined set of the hardware resources of the physical computer from usage by the first hypervisor;

launching the second hypervisor to utilize the predetermined set of the hardware resources; and

migrating remaining hardware resources of the physical computer and the virtual machines from being controlled by the first hypervisor to being controlled by the second hypervisor with the first hypervisor running simultaneously with the second hypervisor during the migration, the second hypervisor sharing access to the single path I/O device with the first hypervisor in different time slots, the first hypervisor and the second hypervisor using a mutex to coordinate the shared access to the single-path I/O device.

2. The computer-implemented method of claim 1, wherein the step of migrating includes repeatedly performing the following steps until all the virtual machines are migrated from the first hypervisor to the second hypervisor:

removing at least a part of the remaining hardware resources from usage by the first hypervisor;

adding said removed part of the remaining hardware resources to usage by the second hypervisor; and

migrating at least one of the virtual machines from the first hypervisor to the second hypervisor.

3. The computer-implemented method of claim 1, further comprising shutting down the first hypervisor after all the virtual machines are migrated from the first hypervisor to the second hypervisor.

4. The computer-implemented method of claim 1, wherein the hardware resources include a plurality of central processing units, and migrating the hardware resources includes migrating a selected one of the central processing units from the first hypervisor to the second hypervisor.

5. The computer-implemented method of claim 1, wherein the hardware resources include a single central processing unit, and migrating the hardware resources includes having the second hypervisor share the central processing unit with the first hypervisor in different time slots.

6. The computer-implemented method of claim 1, wherein the hardware resources include a multi-path input/output (I/O) device with a plurality of I/O paths, and migrating the hardware resources include migrating a selected one of the paths of the I/O device from the first hypervisor to the second hypervisor.

7. The computer-implemented method of claim 1, wherein the hardware resources include a memory device including a plurality of memory pages, and migrating the hardware resources include having the first hypervisor and the second hypervisor control separate pages of the memory device.

8. The computer-implemented method of claim 1, wherein migrating the virtual machines from the first hypervisor to the second hypervisor comprises having the virtual machines run on the second hypervisor without copying data in a virtual machine memory of the virtual machines during the migration.

9. The computer-implemented method of claim 1, wherein the virtualized system is a non-hosted virtualized system.

10. The computer-implemented method of claim 1, wherein the virtualized system is a hosted virtualized system.

11. The computer-implemented method of claim 1, wherein the virtualized system is a para-virtualized system.

12. A computer readable medium storing a computer program product configured to perform a computer-implemented method for migrating control of hardware resources of a physical computer from a first hypervisor to a second hypervisor and installing the second hypervisor on the physical computer, in a virtualized computer system running one or more virtual machines on the first hypervisor, the first hypervisor and the second hypervisor configured to virtualize the hardware resources of the physical computer, the hardware resources including a single-path input/output (I/O) device with a single I/O path, the method comprising:

removing a predetermined set of the hardware resources from usage by the first hypervisor;

launching the second hypervisor to utilize the predetermined set of the hardware resources; and

migrating remaining hardware resources of the physical computer and the virtual machines from being controlled by the first hypervisor to being controlled by the second hypervisor with the first hypervisor running simultaneously with the second hypervisor during the migration, the second hypervisor sharing access to the single path I/O device with the first hypervisor in different time slots, the first hypervisor and the second hypervisor using a mutex to coordinate the shared access to the single-path I/O device.

13. The computer readable medium ‘

of claim 12, wherein the step of migrating includes repeatedly performing the following steps until all the virtual machines are migrated from the first hypervisor to the second hypervisor;

removing at least a part of the remaining hardware resources from usage by the first hypervisor;

adding said removed part of the remaining hardware resources to usage by the second hypervisor; and

migrating at least one of the virtual machines from the first hypervisor to the second hypervisor.

14. The computer readable medium of claim 12, wherein the method further comprises shutting down the first hypervisor after all the virtual machines are migrated from the first hypervisor to the second hypervisor.

15. The computer readable medium of claim 12, wherein the hardware resources include a plurality of central processing units, and migrating the hardware resources include migrating a selected one of the central processing units from the first hypervisor to the second hypervisor.

16. The computer readable medium of claim 12, wherein the hardware resources include a single central processing unit, and migrating the hardware resources include having the second hypervisor share the central processing unit with the first hypervisor in different time slots.

17. The computer readable medium of claim 12, wherein the hardware resources include a multi-path input/output (I/O) device with a plurality of I/O paths, and migrating the hardware resources include migrating a selected one of the paths of the I/O device from the first hypervisor to the second hypervisor.

18. The computer readable medium of claim 12, wherein the hardware resources include a memory device including a plurality of memory pages, and migrating the hardware resources include having the first hypervisor and the second hypervisor control separate pages of the memory device.

19. The computer readable medium of claim 12, wherein migrating the virtual machines from the first hypervisor to the second hypervisor comprises having the virtual machines run on the second hypervisor without copying data in a virtual machine memory of the virtual machines during the migration.

20. A virtualized computer system running one or more virtual machines on a first hypervisor virtualizing hardware resources of a physical computer, the physical computer including a storage device storing computer instructions configured to perform a computer-implemented method of migrating control of the hardware resources of the physical computer from the first hypervisor to a second hypervisor and installing the second hypervisor on the physical computer, the physical computer further including one or more central processing units for executing the computer instructions, the first hypervisor and the second hypervisor being configured to virtualize the hardware resources of the physical computer, the hardware resources including a single-path input/output (I/O) device with a single I/O path, the method comprising:

removing a predetermined set of the hardware resources from usage by the first hypervisor;

launching the second hypervisor to utilize the predetermined set of the hardware resources; and

migrating remaining hardware resources of the physical computer and the virtual machines from being controlled by the first hypervisor to being controlled by the second hypervisor with the first hypervisor running simultaneously with the second hypervisor during the migration, the second hypervisor sharing access to the single path I/O device with the first hypervisor in different time slots, the first hypervisor and the second hypervisor using a mutex to coordinate the shared access to the single-path I/O device.

21. The virtualized computer system of claim 20, wherein the step of migrating includes repeatedly performing the following steps until all the virtual machines are migrated from the first hypervisor to the second hypervisor:

removing at least a part of the remaining hardware resources from usage by the first hypervisor;

adding said removed part of the remaining hardware resources to usage by the second hypervisor; and

migrating at least one of the virtual machines from the first hypervisor to the second hypervisor.

22. The virtualized computer system of claim 20, wherein the method further comprises shutting down the first hypervisor after all the virtual machines are migrated from the first hypervisor to the second hypervisor.

23. The virtualized computer system of claim 20, wherein the hardware resources include a plurality of central processing units, and migrating the hardware resources includes migrating a selected one of the central processing units from the first hypervisor to the second hypervisor.

24. The virtualized computer system of claim 20, wherein the hardware resources include a single central processing unit, and migrating the hardware resources includes having the second hypervisor share the central processing unit with the first hypervisor in different time slots.

25. The virtualized computer system of claim 20, wherein the hardware resources include a multi-path input/output (I/O) device with a plurality of I/O paths, and migrating the hardware resources include migrating a selected one of the paths of the I/O device from the first hypervisor to the second hypervisor.

26. The virtualized computer system of claim 20, wherein the hardware resources include a memory device including a plurality of memory pages, and migrating the hardware resources includes having the first hypervisor and the second hypervisor control separate pages of the memory device.

27. The virtualized computer system of claim 20, wherein migrating the virtual machines from the first hypervisor to the second hypervisor comprises having the virtual machines run on the second hypervisor without copying data in a virtual machine memory of the virtual machines during the migration.

28. In a virtualized computer system running one or more virtual machines on a first hypervisor virtualizing hardware resources of a physical computer, a computer-implemented method of changing control of the hardware resources from the first hypervisor to a second hypervisor, the hardware resources including a single-path input/output (I/O) device with a single I/O path, the method comprising:

removing a predetermined set of the hardware resources of the physical computer from usage by the first hypervisor;

running the second hypervisor to utilize the predetermined set of the hardware resources; and

migrating remaining hardware resources of the physical computer and the virtual machines from being controlled by the first hypervisor to being controlled by the second hypervisor with the first hypervisor running simultaneously with the second hypervisor during the migration, the second hypervisor sharing access to the single path I/O device with the first hypervisor in different time slots, the first hypervisor and the second hypervisor using a mutex to coordinate the shared access to the single-path I/O device.

29. A computer readable medium storing a computer program product configured to perform a computer-implemented method for migrating control of hardware resources of a physical computer from a first hypervisor to a second hypervisor in a virtualized computer system running one or more virtual machines on the first hypervisor, the first hypervisor and the second hypervisor being configured to virtualize the hardware resources of the physical computer, the hardware resources including a single-path input/output (I/O) device with a single I/O path, the method comprising:

removing a predetermined set of the hardware resources of the physical computer from usage by the first hypervisor;

running the second hypervisor to utilize the predetermined set of the hardware resources; and

migrating remaining hardware resources of the physical computer and the virtual machines from being controlled by the first hypervisor to being controlled by the second hypervisor with the first hypervisor running simultaneously with the second hypervisor during the migration, the second hypervisor sharing access to the single path I/O device with the first hypervisor in different time slots, the first hypervisor and the second hypervisor using a mutex to coordinate the shared access to the single-path I/O device.