Systems, methods, and devices for dynamic resource monitoring and allocation in a cluster system

• US 9,647,955 B2
• Filed: 04/21/2016
• Issued: 05/09/2017
• Est. Priority Date: 06/28/2013
• Status: Active Grant

• Current Assignee: Pepperdata, Inc.
• Original Assignee: Pepperdata, Inc.
• Inventors: Suchter, Sean Andrew, Kim, Kimoon, Chang, Choongsoon, Banachowski, Scott Alexander, Hay, Judith A., Carson, Jr., Charles C.
• Primary Examiner(s): Nguyen, Quang N
• Application Number:
  US15/135,447
• Publication Number:
  US 20160373370A1
• Time in Patent Office: 383 Days
• Field of Search: 709217, 709224, 709203, 709206, 709219, 709223, 709226, 709228, 709231, 709232, 709238, 718 1, 718100, 718102, 718104, 707105
• CPC Class Codes: G06F 2209/508 Monitor
• CPC Class Codes: G06F 9/5038 considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration scheduling strategies G06F9/4881 and subgroups
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451
• CPC Class Codes: H04L 41/24 using dedicated network management hardware
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 43/0876 Network utilisation, e.g. volume of load or congestion level
• CPC Class Codes: H04L 47/70 Admission control; Resource allocation
• CPC Class Codes: H04L 47/83 based on usage prediction
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

In an embodiment, the systems, methods, and devices disclosed herein comprise a computer resource monitoring and allocation system. In an embodiment, the resource monitoring and allocation system can be configured to allocate computer resources that are available on various nodes of a cluster to specific jobs and/or sub-jobs and/or tasks and/or processes.

Claims

1. A hadoop computer cluster comprising:

one or more processors of a master node, wherein the master node comprises a supervisor controller; and

one or more processors of a plurality of computing system nodes, the one or more processors of the plurality of computing system nodes configured to perform computing processes on received tasks, wherein each computing system node comprises an agent controller,

wherein each agent controller is configured to monitor system resources utilization by one or more tasks of the respective computing system node and to transmit information describing the monitored system resources utilization to the supervisor controller,

wherein the supervisor controller is configured to assign an additional task to a first computing system node based on determining that the utilization of electronic random access memory of the first computing system node is below a first threshold level, the determination based at least in part on the information describing the monitored system resources utilization, transmitted from the agent controller of the first computing system node,

wherein the supervisor controller is further configured to reduce the use of the electronic random access memory of the second computing system node by a task assigned to the second computing system node based on determining that utilization of the electronic random access memory of the second computing system node is above a second threshold level, the determination based at least in part on the information describing the monitored system resources utilization, transmitted from the agent controller of the second computing system node, and

wherein the master node and the plurality of computing system nodes include a computer processor and an electronic storage medium.

2. The hadoop computer cluster of claim 1, wherein the transmission of information from an agent controller to the supervisor controller describing the monitored system resource utilization is periodic.

3. The hadoop computer cluster of claim 2, wherein the periodic transmission of information from an agent controller to the supervisor controller describing the monitored system resource utilization comprises transmitting in substantially real-time.

4. The hadoop computer cluster of claim 2, wherein each agent controller periodically transmits the monitored system resources utilization of the respective computing system node to the supervisor controller once every 1 to 5 seconds.

5. The hadoop computer cluster of claim 1, wherein reducing the use of the electronic random access memory of the second computing system node by a task assigned to the second computing system node comprises killing that task assigned to the second computing system node.

6. The hadoop computer cluster of claim 1, wherein each computing system node is configured to perform computing processes on a received task corresponding to a container associated with the respective computing system node, wherein each container corresponds to set of resources on the respective computing system node for processing one or more tasks.

7. The hadoop computer cluster of claim 6, wherein the supervisor controller is configured to assign an additional task to the first computing system node through allocating one or multiple additional containers to the first computing system node.

8. The hadoop computer cluster of claim 6, wherein reducing the use of the electronic random access memory of the second computing system node by a task assigned to the second computing system node comprises killing the container associated with that task assigned to the second computing system node.

9. The hadoop computer cluster of claim 6, wherein the master node further comprises a job tracker configured to assign tasks to the plurality of computing system nodes based on the number of containers available at a particular computing system node.

10. The hadoop computer cluster of claim 6, wherein the supervisor controller assigns the additional task to the first computing system node further after having determined that no containers are available at the first computing system node.

11. A supervisor controller configured to dynamically manage assignment of tasks in a hadoop computer cluster, the supervisor controller comprising:

a management controller interface configured to communicate with a management controller to access data representing an assignment of a plurality of tasks across a plurality of computer system nodes in the computer cluster;

an agent controller interface configured to communicate with an agent controller operating on a first computing system node to receive information describing utilization of system resources on the first computing system node, wherein the agent controller interface is further configured to communicate with an agent controller operating on a second computing system node to receive information describing utilization of system resources on the second computing system node; and

a system resource allocation engine configured to assign an additional task to the first computing system node based on determining that the utilization of electronic random access memory of the first computing system node is below a first threshold level, the determination based at least in part on the information describing utilization of system resources on the first computing system node,

wherein one or more computer processors and one or more electronic storage medium are configured to operate the supervisor controller, and

wherein the system resource allocation engine is further configured to reduce the use of electronic random access memory of the second computing system node by a task assigned to the second computing system node based on determining that the utilization of the electronic random access memory of the second computing system node is above a second threshold level, the determination based at least in part on the information describing utilization of system resources on the second computing system node.

12. The supervisor controller of claim 11, wherein each agent controller transmits information describing utilization of system resources of the respective computing system node to the supervisor controller periodically.

13. The supervisor controller of claim 12, wherein the periodic transmission of information from an agent controller to the supervisor controller describing utilization of system resources comprises transmitting in substantially real-time.

14. The supervisor controller of claim 12, wherein each agent controller periodically transmits information describing utilization of system resources of the respective computing system node to the supervisor controller once every 1 to 5 seconds.

15. The supervisor controller of claim 11, wherein reducing the use of the electronic random access memory of the second computing system node by a task assigned to the second computing system node comprises killing that task assigned to the second computing system node.

16. The supervisor controller of claim 11, wherein each computing system node is configured to perform computing processes on a received task within a container on the respective computing system node, wherein each container corresponds to set of resources on the respective computing system node for processing one or more tasks.

17. The supervisor controller of claim 16, wherein the supervisor controller is configured to assign an additional task to the first computing system node through allocating one or multiple additional containers to the first computing system node.

18. The supervisor controller of claim 16, wherein reducing the use of the electronic random access memory of the second computing system node by a task assigned to the second computing system node comprises killing the container associated with that task assigned to the second computing system node.

19. The supervisor controller of claim 16, wherein the master node further comprises a job tracker configured to assign tasks to the plurality of computing system nodes based on the number of containers available at a particular computing system node.

20. The supervisor controller of claim 16, wherein the supervisor controller assigns the additional task to the first computing system node further after having determined that no containers are available at the first computing system node.

21. A supervisor controller configured to dynamically manage assignment of tasks in a hadoop computer cluster, the supervisor controller comprising:

a management controller interface configured to communicate with a management controller to access data representing an assignment of a plurality of tasks across a plurality of computer system nodes in the computer cluster;

an agent controller interface configured to communicate with an agent controller operating on a first computing system node to receive information describing utilization of system resources on the first computing system node, wherein the agent controller interface is further configured to communicate with an agent controller operating on a second computing system node to receive information describing utilization of system resources on the second computing system node; and

a system resource allocation engine configured to assign an additional task to the first computing system node based on determining that the utilization of a CPU of the first computing system node is below a first threshold level, the determination based at least in part on the information describing utilization of system resources on the first computing system node,

wherein one or more computer processors and one or more electronic storage medium are configured to operate the supervisor controller, and

wherein the system resource allocation engine is further configured to reduce the use of a CPU of the second computing system node by a task assigned to the second computing system node based on determining that the utilization of the CPU of the second computing system node is above a second threshold level, the determination based at least in part on the information describing utilization of system resources on the second computing system node.

22. The supervisor controller of claim 21, wherein the transmission of information from an agent controller to the supervisor controller describing the monitored system resource utilization is periodic.

23. The supervisor controller of claim 22, wherein the periodic transmission of information from an agent controller to the supervisor controller describing the monitored system resource utilization comprises transmitting in substantially real-time.

24. The supervisor controller of claim 22, wherein each agent controller periodically transmits the monitored system resources utilization of the respective computing system node to the supervisor controller once every 1 to 5 seconds.

25. The supervisor controller of claim 21, wherein reducing the use of the CPU of the second computing system node by a task assigned to the second computing system node comprises killing that task assigned to the second computing system node.

26. The supervisor controller of claim 21, wherein each computing system node is configured to perform computing processes on a received task corresponding to a container associated with the respective computing system node, wherein each container corresponds to set of resources on the respective computing system node for processing one or more tasks.

27. The supervisor controller of claim 26, wherein the supervisor controller is configured to assign an additional task to the first computing system node through allocating one or multiple additional containers to the first computing system node.

28. The supervisor controller of claim 26, wherein reducing the use of the CPU of the second computing system node by a task assigned to the second computing system node comprises killing the container associated with that task assigned to the second computing system node.

29. The supervisor controller of claim 26, wherein the master node further comprises a job tracker configured to assign tasks to the plurality of computing system nodes based on the number of containers available at a particular computing system node.

30. The supervisor controller of claim 26, wherein the supervisor controller assigns the additional task to the first computing system node further after having determined that no containers are available at the first computing system node.