Booting virtual machine instances in a distributed data processing architecture

• US 10,346,188 B1
• Filed: 06/13/2014
• Issued: 07/09/2019
• Est. Priority Date: 06/13/2014
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Inventors: Christensen, Aaron, Kruck, Joshua D
• Primary Examiner(s): Zhen, Wei Y
• Assistant Examiner(s): Luu, Binh
• Application Number:
  US14/303,904
• Time in Patent Office: 1,852 Days
• Field of Search: None
• CPC Class Codes: G06F 9/4401 Bootstrapping security arrangements therefor G06F21/57
• CPC Class Codes: G06F 9/445 Program loading or initiating bootstrapping G06F9/4401; security arrangements for program loading or initiating G06F21/57
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/5033 considering data affinity
• CPC Class Codes: G06F 9/5077 Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455
• CPC Class Codes: H04L 47/70 Admission control; Resource allocation
• CPC Class Codes: H04L 47/83 based on usage prediction

Abstract

VMs are booted in a big data framework within the context of a cluster of computing and storage devices. The big data framework comprises a distributed, location aware file system and a cluster resource manager that assigns computing resources. VM images are stored as data in the distributed file system. Computing resources and hosts are allocated to specific VMs. The allocated hosts are within given levels of proximity to target data. VMs are booted and run on the hosts, and applications are run on the VMs, processing target data in the distributed file system. Prior to booting a given VM, a stored image can be converted between formats. This enables dynamically determining the VM format at boot time, based on what is supported by the hypervisor available on the target host.

Claims

1. A method comprising:

storing a plurality of images in a distributed file system, wherein

the distributed file system comprises a plurality of storage units,

each of the plurality of images is an image of one of a plurality of virtual machines (VMs),

a target data set is associated with a first VM of the plurality of VMs,

the target data set comprises one or more redundant copies of data associated with the first VM, and

the target data set is distributed across a plurality of the plurality of storage units;

allocating one or more computing resources, from an available pool of computing resources, to the first VM;

identifying at least two storage units in which the target data set is stored, wherein

the at least two storage units are identified from among the plurality of storage units, and

each of the at least two storage units stores a portion of the target data set but not all of the target data set;

selecting a corresponding host that has an acceptable level of physical proximity to the at least two storage units, wherein

the corresponding host is selected from a plurality of hosts, and

the acceptable level of physical proximity is based, at least in part, on one or more requirements of one or more applications that are to be executed on the first VM;

assigning the first VM to the corresponding host;

booting the first VM on the corresponding host, wherein

booting the first VM comprises loading the one or more applications on the first VM; and

executing the one or more applications on the first VM, wherein

the executing the one or more applications comprises processing the target data set by accessing both of the at least two storage units in which the target data set is stored.

2. The method of claim 1 further comprising:

prior to the booting the first VM, converting an image of the first VM from a first format in which the image is stored to a second format; and

booting the first VM in the second format on the corresponding host, wherein a hypervisor on the corresponding host supports VMs in the second format.

3. The method of claim 2 further comprising:

dynamically determining the second format at boot time, wherein

the dynamically determining comprises

identifying one or more formats of images of VMs supported by the hypervisor on the corresponding host.

4. The method of claim 1 wherein the booting the first VM further comprises:

booting the first VM in response to one or more instructions received from a user.

5. The method of claim 1 further comprising:

shutting down the first VM executing on the corresponding host in response to an event from a group of events comprising at least one of the following;

the first VM attempting to utilize more resources than it was allocated;

a shutdown command of an operating system running on the first VM;

or

an application programming interface (API) shutdown.

6. The method of claim 1 further comprising:

performing a given task on the target data set, wherein

the performing comprises booting the first VM multiple times.

7. The method of claim 1, further comprising

booting one or more VMs on the corresponding host;

and

executing a plurality of applications on the one or more VMs, wherein

each application of the plurality of applications processes the target data set differently from at least one other application of the plurality of applications.

8. The method of claim 1, wherein the storing the plurality of images of VMs in the distributed file system further comprises:

performing a backup of a virtualization platform, using the distributed file system as a storage target for a backup image.

9. The method of claim 1 further comprising:

tracking lifecycle events of the first VM, using an analytics tool native to a big data framework; and

providing notifications of the lifecycle events.

10. The method of claim 1, wherein

the selecting the corresponding host is further based on one or more operational characteristics of the corresponding host, wherein

the one or more operational characteristics comprise at least one of:

communication bandwidth between the at least two storage units and the corresponding host,

processing capabilities of each of the corresponding host,

availability of one or more potential hosts from the plurality of hosts,

a characteristic of the one or more applications, and

a characteristic of the target data set.

11. The method of claim 10, wherein

the selecting the corresponding host is further based, at least in part, on a level of performance provided by the corresponding host.

12. The method of claim 1, further comprising:

storing one or more images of VMs in a dedicated third party image management application; and

prior to the booting the first VM on the corresponding host, transferring an image of the first VM to the corresponding host.

13. A non-transitory computer-readable storage medium storing program instructions executable to:

store a plurality of images in a distributed file system, wherein

the distributed file system comprises a plurality of storage units,

each of the plurality of images is an image of one of a plurality of virtual machines (VMs),

a target data set is associated with a first VM of the plurality of VMs,

the target data set comprises one or more redundant copies of data associated with the first VM, and

the target data set is distributed across a plurality of the plurality of storage units;

allocate one or more computing resources, from an available pool of computing resources, to the first VM;

identify at least two storage units in which the target data set is stored, wherein

the at least two storage units are identified from among the plurality of storage units, and

each of the at least two storage units stores a portion of the target data set but not all of the target data set;

select a corresponding host that has an acceptable level of physical proximity to the at least two storage units, wherein

the corresponding host is selected from a plurality of hosts, and

the acceptable level of physical proximity is based, at least in part, on one or more requirements of one or more applications that are to be executed on the first VM;

assign the first VM to the corresponding host;

boot the first VM on the corresponding host, wherein

booting the first VM comprises loading the one or more applications on the first VM; and

execute the one or more applications on the first VM, wherein

executing the one or more applications comprises processing the target data set by accessing both of the at least two storage units in which the target data set is stored.

14. The non-transitory computer-readable storage medium of claim 13, wherein the instructions are further executable to:

prior to booting the first VM, convert an image of the first VM from a first format in which the image is stored to a second format; and

boot the first VM in the second format on the corresponding host, wherein a hypervisor on the corresponding host supports VMs in the second format.

15. The non-transitory computer-readable storage medium of claim 14, wherein the instructions are further executable to:

dynamically determine the second format at boot time, wherein

dynamically determining comprises

identifying one or more formats of images of VMs supported by the hypervisor on the corresponding host.

16. The non-transitory computer-readable storage medium of claim 13, wherein the instructions are further executable to:

shut down the first VM executing on the corresponding host in response to an event from a group of events comprising at least one of the following;

the first VM attempting to utilize more resources than it was allocated;

a shutdown command of an operating system running on the first VM;

or

an application programming interface (API) shutdown.

17. The non-transitory computer-readable storage medium of claim 13, wherein the instructions are further executable to:

perform a given task on the target data set, wherein

performing the given task comprises booting the first VM multiple times.

18. The non-transitory computer-readable storage medium of claim 13, wherein the instructions are further executable to:

boot one or more VMs on the corresponding host; and

execute a plurality of applications on the one or more VMs, wherein

each application of the plurality of applications processes the target data set differently from at least one other application of the plurality of applications.

19. The non-transitory computer-readable storage medium of claim 13, wherein storing the plurality of images of VMs in the distributed file system further comprises:

performing a backup of a virtualization platform, using the distributed file system as a storage target for a backup image.

20. The non-transitory computer-readable storage medium of claim 13, wherein the instructions are further executable to:

track lifecycle events of the first VM, using an analytics tool native to a big data framework; and

provide notifications of the lifecycle events.

21. The non-transitory computer-readable storage medium of claim 13, wherein

selecting the corresponding host is further based on one or more operational characteristics of the corresponding host, wherein

the one or more operational characteristics comprise at least one of:

communication bandwidth between the at least two storage units and the corresponding host,

processing capabilities of each of the corresponding host,

availability of one or more potential hosts from the plurality of hosts,

a characteristic of the one or more applications, and

a characteristic of the target data set.

22. A system comprising:

one or more processors; and

a memory coupled to the one or more processors, wherein the memory stores program instructions executable by the one or more processors to;

store a plurality of images in a distributed file system, wherein

the distributed file system comprises a plurality of storage units,

each of the plurality of images is an image of one of a plurality of virtual machines (VMs),

a target data set is associated with a first VM of the plurality of VMs,

the target data set comprises one or more redundant copies of data associated with the first VM, and

the target data set is distributed across a plurality of the plurality of storage units;

allocate one or more computing resources, from an available pool of computing resources, to the first VM;

identify at least two storage units in which the target data set is stored, wherein

the at least two storage units are identified from among the plurality of storage units, and

each of the at least two storage units stores a portion of the target data set but not all of the target data set;

select a corresponding host that has an acceptable level of physical proximity to the at least two storage units, wherein

the corresponding host is selected from a plurality of hosts, and

the acceptable level of physical proximity is based, at least in part, on one or more requirements of one or more applications that are to be executed on the first VM;

assign the first VM to the corresponding host;

boot the first VM on the corresponding host, wherein

booting the first VM comprises loading the one or more applications on the first VM; and

execute the one or more applications on the first VM, wherein

executing the one or more applications comprises processing the target data set by accessing both of the at least two storage units in which the target data set is stored.