MULTI-TENANT DISTRIBUTED COMPUTING AND DATABASE

• US 20150121371A1
• Filed: 07/11/2014
• Published: 04/30/2015
• Est. Priority Date: 10/25/2013
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: GUMMARAJU, Jayanth, LU, Yunshan, MAGDON-ISMAIL, Tariq
• Granted Patent: US 10,642,800 B2
• Time in Patent Office: Days
• US Class Current: 718/1
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/188 Virtual file systems
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 2009/45562 Creating, deleting, cloning virtual machine instances
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects

Abstract

A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Deployments of a distributed computing application, such as Hadoop, may be executed concurrently with a distributed database application, such as HBase, using a shared instance of a distributed filesystem, or in other cases, multiple instances of the distributed filesystem. Computing resources allocated to region server nodes executing as VMs may be isolated from compute VMs of the distributed computing application, as well as from data nodes executing as VMs of the distributed filesystem.

Claims

1. A method for executing a distributed computing application within a virtualized computing environment for a plurality of tenants, the method comprising:

instantiating a first plurality of virtual machines (VMs) on a plurality of hosts to form a first distributed filesystem accessible by a plurality of compute VMs, wherein each compute VM is configured to process a portion of an input data set stored in the first distributed filesystem; and

instantiating a second plurality of VMs on the plurality of hosts to form a second distributed filesystem accessible by a plurality of region server nodes associated with a distributed database application, wherein each region server node is configured to serve a portion of a data table stored in the second distributed filesystem.

2. The method of claim 1, wherein instantiating the second plurality of VMs on the plurality of hosts to form the second distributed filesystem comprises:

instantiating a first VM comprising a first region server node and a first data node executing on the same VM, wherein the first data node is configured to store a portion of the data table associated with the first region server node.

3. The method of claim 2, wherein the plurality of compute VMs comprises a first resource pool associated with the distributed computing application, and the second plurality of VMs comprises a second resource pool associated with the distributed database application, wherein the second resource pool has a higher priority for computing resources than the first resource pool.

4. The method of claim 1, wherein instantiating the second plurality of VMs on the plurality of hosts to form the second distributed filesystem comprises:

instantiating a first VM comprising a first region server node; and

instantiating a second VM comprising a first data node configured to store a portion of the data table associated with the first region server node, wherein the first VM and the second VM are executing on the same host.

5. The method of claim 4, wherein the plurality of compute VMs comprises a first resource pool associated with the distributed computing application, wherein the first VM is a member of a second resource pool associated with the distributed database application, wherein the second VM is a member of a third resource pool associated with the second distributed filesystem.

6. The method of claim 4, further comprising:

responsive to an indication to expand the distributed database application, instantiating a third VM comprising a second region server node configured to store a portion of the data table associated with the second region server node within the second VM; and

responsive to an indication to shrink the distributed database application, powering off the first VM comprising the first region server node.

7. The method of claim 1, further comprising:

executing a HBase query on at least one of the plurality of region server nodes while concurrently executing a MapReduce job on the plurality of compute VMs.

8. A non-transitory computer-readable storage medium comprising instructions that, when executed in a computing device, execute a distributed computing application within a virtualized computing environment for a plurality of tenants, by performing the steps of:

instantiating a first plurality of virtual machines (VMs) on a plurality of hosts to form a first distributed filesystem accessible by a plurality of compute VMs, wherein each compute VM is configured to process a portion of an input data set stored in the first distributed filesystem; and

instantiating a second plurality of VMs on the plurality of hosts to form a second distributed filesystem accessible by a plurality of region server nodes associated with a distributed database application, wherein each region server node is configured to serve a portion of a data table stored in the second distributed filesystem.

9. The non-transitory computer-readable storage medium of claim 8, wherein the step of instantiating the second plurality of VMs on the plurality of hosts to form the second distributed filesystem comprises:

instantiating a first VM comprising a first region server node and a first data node executing on the same VM, wherein the first data node is configured to store a portion of the data table associated with the first region server node.

10. The non-transitory computer-readable storage medium of claim 9, wherein the plurality of compute VMs comprises a first resource pool associated with the distributed computing application, and the second plurality of VMs comprises a second resource pool associated with the distributed database application, wherein the second resource pool has a higher priority for computing resources than the first resource pool.

11. The non-transitory computer-readable storage medium of claim 8, wherein the step of instantiating the second plurality of VMs on the plurality of hosts to form the second distributed filesystem comprises:

instantiating a first VM comprising a first region server node; and

instantiating a second VM comprising a first data node configured to store a portion of the data table associated with the first region server node, wherein the first VM and the second VM are executing on the same host.

12. The non-transitory computer-readable storage medium of claim 11, wherein the plurality of compute VMs comprises a first resource pool associated with the distributed computing application, wherein the first VM is a member of a second resource pool associated with the distributed database application, wherein the second VM is a member of a third resource pool associated with the second distributed filesystem.

13. The non-transitory computer-readable storage medium of claim 11, further comprising:

responsive to an indication to expand the distributed database application, instantiating a third VM comprising a second region server node; and

responsive to an indication to shrink the distributed database application, powering off the first VM comprise the first region server node.

14. The non-transitory computer-readable storage medium of claim 8, further comprising:

executing a HBase query on at least one of the plurality of region server nodes while concurrently executing a MapReduce job on the plurality of compute VMs.

15. A host computer system for executing a distributed computing application within a virtualized computing environment for a plurality of tenants, the host computer system comprising:

a storage device having a first virtual disk and a second virtual disk;

a processor programmed to carry out the steps of;

executing a first virtual machine (VM) on the host computer system, wherein the first VM includes the first virtual disk and is part of a first distributed filesystem accessible by a plurality of compute VMs;

executing, on the host computer system, a second VM of the plurality of compute VMs, wherein the second VM is configured to process a portion of an input data set stored in the first virtual disk; and

executing a third VM on the host computer system, wherein the third VM includes the second virtual disk and is part of a second distributed filesystem accessible by a plurality of region server nodes associated with a distributed database application, wherein each region server node is configured to serve a portion of a data table stored in the second virtual disk.

16. The host computer system of claim 15, wherein the third VM comprise a first region server node and a first data node executing on the same VM, wherein the first data node is configured to store a portion of the data table associated with the first region server node.

17. The host computer system of claim 16, wherein the second VM is a member of a first resource pool associated with the distributed computing application, and the third VM is a member of a second resource pool associated with the distributed database application, wherein the second resource pool has a higher priority for computing resources of the host computer system than the first resource pool.

18. The host computer system of claim 15, wherein the processor is further programmed to carry out the steps of:

executing a fourth VM comprising a first region server node, wherein the third VM comprises a first data node configured to store a portion of the data table associated with the first region server node.

19. The host computer system of claim 18, wherein the second VM comprises a first resource pool associated with the distributed computing application, wherein the fourth VM is a member of a second resource pool associated with the distributed database application, wherein the third VM is a member of a third resource pool associated with the second distributed filesystem.

20. The host computer system of claim 18, wherein the processor is further programmed to carry out the steps of:

responsive to an indication to expand the distributed database application, instantiating a fifth VM comprising a second region server node configured to store a portion of the data table associated with the second region server node within the third VM; and

responsive to an indication to shrink the distributed database application, powering off the third VM comprising the first region server node.