MULTI-TENANT PRODUCTION AND TEST DEPLOYMENTS OF HADOOP

• US 20150120791A1
• Filed: 10/24/2013
• Published: 04/30/2015
• Est. Priority Date: 10/24/2013
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: GUMMARAJU, Jayanth, LU, Yunshan, CHEVERESAN, Razvan
• Granted Patent: US 10,216,758 B2
• Time in Patent Office: Days
• US Class Current: 707/823
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 2009/45575 Starting, stopping, suspending or resuming virtual machine instances
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects

Abstract

A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Production, test, and development deployments of a Hadoop application may be executed using multiple compute clusters and a shared instance of a distributed filesystem, or in other cases, multiple instances of the distributed filesystem. Data nodes executing as virtual machines (VMs) for test and development deployments can be linked clones of data nodes executing as VMs for a production deployment to reduce duplicated data and provide a shared storage space.

Claims

1. A method for executing a distributed computing application within a virtualized computing environment for a plurality of tenants, the method comprising:

instantiating a first plurality of virtual machines (VMs) on a plurality of hosts to form a first distributed filesystem, wherein at least one VM of the first plurality of VMs includes a virtual disk configured to store data blocks;

storing an input data set in the first distributed filesystem, wherein the input data set comprises a plurality of data blocks, wherein the first distributed filesystem is accessible by a plurality of compute VMs configured to process the input data set; and

instantiating a second plurality of VMs on the plurality of hosts to form a second distributed filesystem storing the same input data set, wherein each instantiated VM of the second plurality of VMs comprises a linked clone that references a virtual disk of a corresponding VM in the first plurality of VMs.

2. The method of claim 1, wherein the virtual disk of a first VM of the first plurality of VMs comprises a base virtual disk storing data blocks and a first delta disk, and wherein the virtual disk of a corresponding second VM of the second plurality of VMs comprises a second delta disk that references the base virtual disk of the first VM.

3. The method of claim 2, further comprising:

instantiating a third plurality of VMs on the plurality of hosts to form a third distributed filesystem storing the same input data set, wherein at least one VM of the third plurality of VMs includes a linked clone that references the base virtual disk of the first VM, not the first delta disk.

4. The method of claim 1, further comprising:

instantiating a third plurality of VMs on the plurality of hosts to form a third distributed filesystem storing the same input data set, wherein at least one VM of the third plurality of VMs includes a linked clone that references the linked clone of a corresponding VM of the second plurality of VMs.

5. The method of claim 1, wherein instantiating the second plurality of VMs on the plurality of hosts to form the second distributed filesystem further comprises:

instantiating a second name node VM associated with the second distributed filesystem, wherein the second name node comprises a linked clone that references a virtual disk of a first name node VM associated with the first distributed filesystem, wherein the virtual disk of the first name node VM comprises location information for each data block of the input data set and a corresponding location within the first plurality of VMs; and

modifying a configuration of the second plurality of VMs to include information associated with the second name node VM.

6. The method of claim 1, wherein the first plurality of VMs are associated with a first resource pool, and the second plurality of VMs are associated with a second resource pool.

7. The method of claim 1, wherein the first distributed filesystem is associated with a production environment for processing the input data set, and the second distributed filesystem is associated with a test environment.

8. The method of claim 1, wherein the plurality of compute VMs comprises a first plurality of compute VMs associated with executing a first application for processing the input data set, and a second plurality of compute VMs associated with executing a second application for processing the input data set.

9. The method of claim 1, wherein the plurality of compute VMs comprises a plurality of container VMs and at least one node manager VM configured to launch the plurality of container VMs for executing tasks on the input data set.

10. A non-transitory computer-readable storage medium comprising instructions that, when executed in a computing device, execute a distributed computing application within a virtualized computing environment for a plurality of tenants, by performing the steps of:

instantiating a first plurality of virtual machines (VMs) on a plurality of hosts to form a first distributed filesystem, wherein at least one VM of the first plurality of VMs includes a virtual disk configured to store data blocks;

storing an input data set in the first distributed filesystem, wherein the input data set comprises a plurality of data blocks, wherein the first distributed filesystem is accessible by a plurality of compute VMs configured to process the input data set; and

instantiating a second plurality of VMs on the plurality of hosts to form a second distributed filesystem storing the same input data set, wherein each instantiated VM of the second plurality of VMs comprises a linked clone that references a virtual disk of a corresponding VM in the first plurality of VMs.

11. The non-transitory computer-readable storage medium of claim 10, wherein the virtual disk of a first VM of the first plurality of VMs comprises a base virtual disk storing data blocks and a first delta disk, and wherein the virtual disk of a corresponding second VM of the second plurality of VMs comprises a second delta disk that references the base virtual disk of the first VM.

12. The non-transitory computer-readable storage medium of claim 11, further comprising the steps of:

instantiating a third plurality of VMs on the plurality of hosts to form a third distributed filesystem storing the same input data set, wherein at least one VM of the third plurality of VMs includes a linked clone that references the base virtual disk of the first VM, not the first delta disk.

13. The non-transitory computer-readable storage medium of claim 10, further comprising the steps of:

instantiating a third plurality of VMs on the plurality of hosts to form a third distributed filesystem storing the same input data set, wherein at least one VM of the third plurality of VMs includes a linked clone that references the linked clone of a corresponding VM of the second plurality of VMs.

14. The non-transitory computer-readable storage medium of claim 10, wherein instantiating the second plurality of VMs on the plurality of hosts to form the second distributed filesystem further comprises the steps of:

instantiating a second name node VM associated with the second distributed filesystem, wherein the second name node comprises a linked clone that references a virtual disk of a first name node VM associated with the first distributed filesystem, wherein the virtual disk of the first name node VM comprises location information for each data block of the input data set and a corresponding location within the first plurality of VMs; and

modifying a configuration of the second plurality of VMs to include information associated with the second name node VM.

15. The non-transitory computer-readable storage medium of claim 10, wherein the first plurality of VMs are associated with a first resource pool, and the second plurality of VMs are associated with a second resource pool.

16. The non-transitory computer-readable storage medium of claim 10, wherein the first distributed filesystem is associated with a production environment for processing the input data set, and the second distributed filesystem is associated with a test environment.

17. The non-transitory computer-readable storage medium of claim 10, wherein the plurality of compute VMs comprises a first plurality of compute VMs associated with executing a first application for processing the input data set, and a second plurality of compute VMs associated with executing a second application for processing the input data set.

18. A computer system having a plurality of hosts executing a plurality of virtual machines (VMs) for executing a distributed computing application within a virtualized computing environment for a plurality of tenants, the computer system comprising:

a memory; and

a processor programmed to carry out the steps of;

instantiating a first plurality of virtual machines (VMs) on a plurality of hosts to form a first distributed filesystem, wherein at least one VM of the first plurality of VMs includes a virtual disk configured to store data blocks;

storing an input data set in the first distributed filesystem, wherein the input data set comprises a plurality of data blocks, wherein the first distributed filesystem is accessible by a plurality of compute VMs configured to process the input data set; and

instantiating a second plurality of VMs on the plurality of hosts to form a second distributed filesystem storing the same input data set, wherein each instantiated VM of the second plurality of VMs comprises a linked clone that references a virtual disk of a corresponding VM in the first plurality of VMs.

19. The computer system of claim 18, wherein the virtual disk of a first VM of the first plurality of VMs comprises a base virtual disk storing data blocks and a first delta disk, and wherein the virtual disk of a corresponding second VM of the second plurality of VMs comprises a second delta disk that references the base virtual disk of the first VM.

20. The computer system of claim 18, wherein the processor is further programmed to carry out the steps of:

instantiating a third plurality of VMs on the plurality of hosts to form a third distributed filesystem storing the same input data set, wherein at least one VM of the third plurality of VMs includes a linked clone that references the linked clone of a corresponding VM of the second plurality of VMs.