Dynamic resource management of a pool of resources for multi-tenant applications based on sample exceution, query type or jobs

• US 10,713,092 B2
• Filed: 01/02/2018
• Issued: 07/14/2020
• Est. Priority Date: 01/02/2018
• Status: Active Grant

• Current Assignee: JP Morgan Chase Bank, N.A. (JP Morgan Chase & Co)
• Original Assignee: JP Morgan Chase Bank, N.A. (JP Morgan Chase & Co)
• Inventors: Gupta, Akhilesh, Addanki, Suman Kumar, Cuddihy, James P., Rajaram, Jay, Mishra, Ratikanta, Aguiling, Michael
• Primary Examiner(s): An, Meng Al T
• Assistant Examiner(s): Chu Joy-Davila, Jorge A
• Application Number:
  US15/860,021
• Publication Number:
  US 20190205173A1
• Time in Patent Office: 924 Days
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 2209/5011 Pool
• CPC Class Codes: G06F 9/5016 the resource being the memory
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources

Abstract

Systems and methods for resource management for multi-tenant applications in a Hadoop cluster are disclosed. In one embodiment, in an information processing device comprising at least one computer processor, a method for resource management for multi-tenant applications in a Hadoop cluster may include: (1) allocating an initial allocation of a resource in a resource pool to a plurality of tenants, each tenant having a workload; (2) determining a memory requirement for each of the plurality of tenants; (3) determining a maximum number of concurrent queries or jobs for each of the plurality of tenants; (4) determining a memory and vcore requirement for each of the plurality of tenants based on the memory requirement and maximum number of concurrent queries or jobs; and (5) allocating the resources to each of the plurality of tenants.

Claims

1. A method for dynamic resource management for multi-tenant applications in a distributed environment of a computational computer cluster that stores and analyzes big data, the method is performed by an information processing device comprising at least one computer processor, the method comprising the steps of:

determining, as each new tenant is onboarded, an initial allocation of resources from a resource pool to a plurality of tenants, each tenant having an anticipated query type or jobs, wherein the resource pool supports different services and the initial allocation of the resources is a percentage of each of the different services based on the anticipated query type or jobs;

wherein the anticipated query type or jobs are associated with application categories comprising at least one of batch, streaming, and archiving;

assigning to each tenant of the plurality of tenants the initial allocation of resources;

determining a baseline tenant memory requirement for each of the plurality of tenants based on a sample execution or an Explain Plan of a query type or jobs;

determining a maximum number of concurrent queries or jobs for each of the plurality of tenants based on an application category requirement for an application executed by the tenant;

executing the anticipated query type or jobs;

monitoring an actual number of the concurrent queries or jobs running, and when the actual number of the concurrent queries is the same as the determined maximum number of concurrent queries and additional queries are waiting;

determining an actual memory requirement and an actual virtual core requirement for each of the plurality of tenants based on the baseline memory requirement and the monitored actual number of the concurrent queries or jobs; and

updating the initial allocation of the resources with additional resources based on the determining until queries are completed; and

allocating the resources to each of the plurality of tenants based on the updated initial allocation.

2. The method of claim 1, wherein the initial allocation is further based on an anticipated number of tenants.

3. The method of claim 1, wherein the resource pool is a dynamic resource pool.

4. The method of claim 1, further comprising:

restricting access to at least one of the resources using an access control list.

5. The method of claim 1, wherein each tenant is guaranteed its initial allocation of resources.

6. A system for dynamic resource management for multi-tenant applications in a distributed environment of a computational computer cluster that stores and analyzes big data, the system comprising an information processing device comprising at least one computer processor that performs the following:

determines, as each new tenant is onboarded, an initial allocation of resources from a resource pool to each of a plurality of tenants, each tenant having an anticipated query type or jobs, wherein the resource pool comprises a plurality of resources supported by different services and the initial allocation of resources is a percentage of each of the different services based on the anticipated query type or jobs;

wherein the anticipated query type or jobs are associated with application categories comprising at least one of batch, streaming, and archiving;

assigns to each of the tenants the initial allocation of resources;

determines a baseline tenant memory requirement for each of the plurality of tenants based on a sample execution or an Explain Plan of a query type or jobs;

determines a maximum number of concurrent queries or jobs for each of the plurality of tenants based on an application category requirement for an application executed by the tenant;

executes the anticipated query type or jobs;

monitors an actual number of the concurrent queries or jobs running, and when the actual number of the concurrent queries is the same as the determined maximum number of concurrent queries and additional queries are waiting;

determines an actual memory requirement and an actual virtual core requirement for each of the plurality of tenants based on the baseline memory requirement and the monitored actual number of concurrent queries or jobs; and

updates the initial allocation of the resources with additional resources based on the determining until queries are completed; and

allocates the resources to each of the plurality of tenants based on the updated initial allocation.

7. The system of claim 6, wherein the initial allocation is further based on and an anticipated query type.

8. The system of claim 6, wherein the resource pool is a dynamic resource pool.

9. The system of claim 6, further comprising:

restricting access to the resource using an access control list.

10. The system of claim 6, wherein access to at least one of the resources is restricted using an access control list.