Migrating virtual machines among computer systems to balance load caused by virtual machines

• US 7,203,944 B1
• Filed: 07/09/2003
• Issued: 04/10/2007
• Est. Priority Date: 07/09/2003
• Status: Expired due to Fees

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Veritas Operating Corporation (Gen Digital Inc.)
• Inventors: van Rietschote, Hans F., Hobbs, Craig W., Saptarshi, Mahesh P.
• Primary Examiner(s): An; Meng-Al T.
• Assistant Examiner(s): Truong; Camquy
• Application Number:
  US10/616,437
• Time in Patent Office: 1,371 Days
• Field of Search: 718/103, 718/104, 718/1
• US Class Current: 718/104
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/4856 resumption being on a different machine, e.g. task migration, virtual machine migration G06F9/5088 takes precedence
• CPC Class Codes: G06F 9/5088 involving task migration

Abstract

A cluster comprises a plurality of computer systems, wherein each of the plurality of computer systems is configured to execute one or more virtual machines. Each of the plurality of computer systems comprises hardware and a plurality of instructions. The plurality of instructions, when executed on the hardware, migrates at least a first virtual machine executing on a first computer system of the plurality of computer systems to a second computer system of the plurality of computer systems. The plurality of instructions migrates the first virtual machine responsive to a first load of the first computer system prior to the migration exceeding a second load of the second computer system prior to the migration.

Claims

1. A cluster comprising a plurality of computer systems, wherein each of the plurality of computer systems is configured to execute one or more virtual machines, each of the plurality of computer systems comprising hardware and a plurality of instructions that, when executed on the hardware, detects that a first load of a first computer system of the plurality of computer systems exceeds a second load of a second computer system of the plurality of computer systems and migrates at least a first virtual machine executing on the first computer system to the second computer system responsive to detecting that the first load exceeds the second load, and wherein the first virtual machine has a corresponding load that is nearest, among loads of the virtual machines executing on the first computer system, to ½

the difference between the first load and the second load.

2. The cluster as recited in claim 1 wherein the first virtual machine executes on the second computer system independent of the first computer system, even if the first virtual machine was initially launched on the first computer system.

3. The cluster as recited in claim 1 wherein the plurality of instructions, when executed on the first computer system, select the second computer system to compare loads.

4. The cluster as recited in claim 3 wherein the plurality of instructions, when executed on the first computer system, randomly select the second computer system from the plurality of computer systems.

5. The cluster as recited in claim 1 wherein the corresponding load of the first virtual machine represents the actual load experienced in executing the first virtual machine on the first computer system.

6. The cluster as recited in claim 1 wherein the corresponding load of the first virtual machine represents a target load programmed for the first virtual machine in the first computer system.

7. The cluster as recited in claim 1 wherein the first virtual machine has a first corresponding load on the first computer system and a second corresponding load on the second computer system, and wherein the first corresponding load differs from the second corresponding load, wherein the first computer system is configured to transmit one or more load factors to the second computer system, and wherein the second computer system is configured to calculate the second corresponding load from the one or more load factors, and wherein the first computer system and the second computer system are configured to exchange the first corresponding load and the second corresponding load to select the first virtual machine for migration.

8. The cluster as recited in claim 1 wherein the corresponding load is calculated as a weighted combination of measurements of usage of two or more resources of the first computer system.

9. The cluster as recited in claim 8 wherein the measurements of usage include an amount of time that the first virtual machine is executing in a central processing unit of the first computer system.

10. The cluster as recited in claim 8 wherein the measurements of usage include an amount of input/output activity generated by the first virtual machine during execution.

11. The cluster as recited in claim 8 wherein the measurements of usage include an amount of memory occupied by the first virtual machine.

12. The cluster as recited in claim 1 wherein each of the plurality of computer systems include a schedule having a plurality of entries, each entry corresponding to a virtual machine to be executed on the respective one of the plurality of computer systems, and wherein migrating the first virtual machine comprises deleting the entry corresponding to the first virtual machine in the schedule of the first computer system and inserting the entry corresponding to the first virtual machine in the schedule of the second computer system.

13. A method comprising:

scheduling one or more virtual machines for execution on hardware comprising a first computer system of a plurality of computer systems;

the first computer system detecting that the first computer system has a first load that exceeds a second load of a second computer system of the plurality of computer systems; and

the first computer system migrating at least a first virtual machine executing on the first computer system to a second computer system of the plurality of computer systems responsive to the detecting, and wherein the first virtual machine has a corresponding load that is nearest, among the virtual machines executing on the first computer system, to ½

the difference between the first load and the second load.

14. The method as recited in claim 13 further comprising executing the first virtual machine on the second computer system independent of the first computer system, even if the first virtual machine was initially launched on the first computer system.

15. The method as recited in claim 13 further comprising:

selecting the second computer system to compare loads; and

selecting the first virtual machine to migrate to the second computer system responsive to the first load exceeding the second load.

16. The method as recited in claim 15 wherein selecting the second computer system is random.

17. The method as recited in claim 16 further comprising each of the plurality of computer systems periodically randomly selecting another one of the plurality of computer systems to compare loads and to potentially migrate virtual machines.

18. The method as recited in claim 13 wherein the first virtual machine has a first corresponding load on the first computer system and a second corresponding load on the second computer system, and wherein the first corresponding load differs from the second corresponding load, the method further comprising:

the first computer system transmitting one or more load factors to the second computer system;

the second computer system calculating the second corresponding load from the one or more load factors; and

the first computer system and the second computer system exchanging the first corresponding load and the second corresponding load to select the first virtual machine for migration.

19. The method as recited in claim 13 further comprising calculating the corresponding load as a weighted combination of measurements of usage of two or more resources of the first computer system.

20. The method as recited in claim 13 wherein each of the plurality of computer systems include a schedule having a plurality of entries, each entry corresponding to a virtual machine to be executed on the respective one of the plurality of computer systems, and wherein migrating the first virtual machine comprises:

deleting the entry corresponding to the first virtual machine in the schedule of the first computer system; and

inserting the entry corresponding to the first virtual machine in the schedule of the second computer system.

21. The computer accessible medium as recited in claim 20 wherein the first virtual machine executes on the second computer system independent of the first computer system during use, even if the first virtual machine was initially launched on the first computer system.

22. A computer accessible medium encoded with a plurality of instructions that, when executed on a first computer system:

select a first virtual machine from one or more virtual machines to be scheduled for execution on the first computer system responsive to a first load of the first computer system exceeding a second load of a second computer system of a plurality of computer systems including the first computer system, and wherein the first virtual machine has a corresponding load that is approximately ½

the difference between the first load and the second load; and

migrate the first virtual machine to the second computer system to be executed on the second computer system.

23. The computer accessible medium as recited in claim 22 wherein the plurality of instructions, when executed, schedule the one or more virtual machines for execution on hardware comprising the first computer system.

24. The computer accessible medium as recited in claim 22 wherein the plurality of instructions, when executed, select the second computer system to compare loads.

25. The computer accessible medium as recited in claim 24 wherein the second computer system is randomly selected from the plurality of computer systems.

26. The computer accessible medium as recited in claim 22 wherein the corresponding load of the first virtual machine represents the actual load experienced in executing the first virtual machine on the first computer system.

27. The computer accessible medium as recited in claim 22 wherein the corresponding load of the first virtual machine represents a target load programmed for the first virtual machine in the first computer system.

28. The computer accessible medium as recited in claim 22 wherein the first virtual machine has a first corresponding load on the first computer system and a second corresponding load on the second computer system, and wherein the first corresponding load differs from the second corresponding load, and wherein the plurality of instructions, when executed:

transmit one or more load factors to the second computer system, wherein the second computer system is configured to calculate the second corresponding load from the one or more load factors; and

exchanges the first corresponding load and the second corresponding load with the second computer system to select the first virtual machine for migration.

29. The computer accessible medium as recited in claim 22 wherein the plurality of instructions, when executed, calculate the corresponding load as a weighted combination of measurements of usage of two or more resources of the first computer system.

30. The computer accessible medium as recited in claim 22 wherein each of the plurality of computer systems include a schedule having a plurality of entries, each entry corresponding to a virtual machine to be executed on the respective one of the plurality of computer systems, and wherein the plurality of instructions migrate the first virtual machine by:

deleting the entry corresponding to the first virtual machine in the schedule of the first computer system; and

inserting the entry corresponding to the first virtual machine in the schedule of the second computer system.

31. The computer accessible medium as recited in claim 30 wherein the plurality of instructions, when executed, calculate the first load as a weighted combination of measurements of usage of two or more resources of the first computer system, and wherein the second load is calculated as a weighted combination of measurements, wherein the weights differ between the first and second computer systems.

32. A computer accessible medium encoded with a plurality of instructions that, when executed on a first computer system:

calculate a first load on the first computer system from one or more load factors, wherein the first load corresponds to a first virtual machine, and wherein the first virtual machine is one of one or more virtual machines to be scheduled for execution on the first computer system;

transmit the one or more load factors from the first computer system to a second computer system, wherein the second computer system is configured to calculate a second load on the second computer system from the one or more load factors, wherein the second load corresponds to the first virtual machine, and wherein the first load differs from the second load; and

migrate the first virtual machine to the second computer system to be executed on the second computer system responsive to a first total load of the first computer system exceeding a second total load of the second computer system, and wherein the first virtual machine has a corresponding load that is nearest, among loads of the virtual machines executing on the first computer system, to ½

the difference between the first total load and the second total load.

33. The computer accessible medium as recited in claim 32 wherein the first virtual machine executes on the second computer system independent of the first computer system during use, even if the first virtual machine was initially launched on the first computer system.

34. The computer accessible medium as recited in claim 32 wherein each of the plurality of computer systems include a schedule having a plurality of entries, each entry corresponding to a virtual machine to be executed on the respective one of the plurality of computer systems, and wherein the plurality of instructions migrate the first virtual machine by:

deleting the entry corresponding to the first virtual machine in the schedule of the first computer system; and

inserting the entry corresponding to the first virtual machine in the schedule of the second computer system.