Systems, methods, and devices for dynamic resource monitoring and allocation in a cluster system

• US 8,849,891 B1
• Filed: 02/28/2014
• Issued: 09/30/2014
• Est. Priority Date: 06/28/2013
• Status: Active Grant

• Current Assignee: Pepperdata, Inc.
• Original Assignee: Pepperdata, Inc.
• Inventors: Suchter, Sean Andrew, Carson, Charles C. Jr., Kim, Kimoon, Chang, Choongsoon, Banachowski, Scott Alexander, Hay, Judith A.
• Primary Examiner(s): Nguyen, Quang N
• Application Number:
  US14/194,406
• Time in Patent Office: 214 Days
• Field of Search: 709/202, 709/217, 709/219, 709/223, 709/224, 718/102, 718/104, 718/106
• US Class Current: 709/202
• CPC Class Codes: G06F 2209/508 Monitor
• CPC Class Codes: G06F 9/5038 considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration scheduling strategies G06F9/4881 and subgroups
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451
• CPC Class Codes: H04L 41/24 using dedicated network management hardware
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 43/0876 Network utilisation, e.g. volume of load or congestion level
• CPC Class Codes: H04L 47/70 Admission control; Resource allocation
• CPC Class Codes: H04L 47/83 based on usage prediction
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

In an embodiment, the systems, methods, and devices disclosed herein comprise a computer resource monitoring and allocation system. In an embodiment, the resource monitoring and allocation system can be configured to allocate computer resources that are available on various nodes of a cluster to specific jobs and/or sub-jobs and/or tasks and/or processes.

Claims

1. A computer cluster comprising:

a management computing device comprising a supervisor controller configured to coordinate processing of a plurality of sub-jobs for a plurality of overall jobs;

a plurality of computer system nodes configured to communicate with the management computing device, and to perform processing of received sub-jobs, the computing system nodes each comprising;

one or more processors configured to perform computing processes on received sub-jobs;

an agent controller comprising;

a monitoring interface configured to monitor utilization by sub-jobs of system resources of a first computing system node; and

a reporting controller configured to transmit the monitored system resources utilization to the supervisor controller in substantially real-time;

wherein the supervisor controller is configured to assign an additional sub-job to the first computing system node based on determining that the utilization of at least one system resource of the first computing system node is below a threshold level, the determining based on the monitored system resources utilization transmitted from the reporting controller to the supervisor controller;

wherein the at least one system resource of the first computing system node is a first electronic random access memory capacity,

wherein the supervisor controller is configured to monitor a second electronic random access memory capacity of a second computing system node,

wherein the assigning by the supervisor controller of the additional sub-job comprises assigning the additional sub-job to the first computing system node based on determining that utilization of the first electronic random access memory capacity is below the threshold level,

wherein the supervisor controller is configured to prevent assignment of additional sub-jobs to the second computing system node based on determining that utilization of the second electronic random access memory capacity is at or above a threshold value,

wherein the additional sub-job requires utilization of the first electronic random access memory capacity that is unused on the first computing system node.

2. The computer cluster of claim 1, wherein the computer cluster is a Hadoop computer cluster.

3. The computer cluster of claim 2, wherein the management computing device comprises a management controller, the management controller comprises a job tracker or a yarn resource manager, wherein the plurality of computer system nodes comprise task trackers or yarn node managers configured to communicate data to the job tracker or the yarn resource manager, the communicated data comprising a number of available slots or containers for processing tasks on a particular computer system node.

4. The computer cluster of claim 3, wherein the supervisor controller is configured to assign the additional sub-job to the first computing system node based on determining that the utilization of the at least one system resource of the computing system node is below the threshold level and based on determining that the number of available slots or containers is zero for the first computing system node.

5. The computer cluster of claim 1, wherein the supervisor controller is configured to assign the additional sub-job to the first computing system node based on determining that the utilization of the at least one system resource of the first computing system node is below the threshold level and based on determining that the utilization of a second system resource of the first computing system node is above a second threshold level, the determining based on the monitored system resources utilization transmitted from the reporting controller to the supervisor controller.

6. The computer cluster of claim 5, wherein the additional sub-job assigned by the supervisor controller requires greater utilization of the at least one system resource as compared to the utilization of the second system resource.

7. The computer cluster of claim 1, wherein the plurality of computer system nodes comprises computing devices having heterogeneous machine types.

8. The computer cluster of claim 7, wherein the first electronic random access memory capacity is larger than the second electronic random access memory capacity, wherein the assigning of the additional sub-job comprises assigning the additional sub-job to the first computing system based on determining that the utilization of the first electronic random access memory capacity is below the threshold level.

9. The computer cluster claim 8, wherein the first computing system node is processing more sub-jobs than the second computing system node.

10. A supervisor controller configured to dynamically manage assignment of job processes in a computer cluster, the supervisor controller comprising:

a management controller interface configured to communicate with a management controller to access data representing an assignment of a plurality of job processes across a plurality of computer system nodes in the computer cluster;

an agent controller interface configured to communicate with an agent controller operating on a first computing system node, the agent controller configured to transmit to the agent controller interface data representing utilization of system resources by the plurality of job processes operating on the first computing system node; and

a system resource allocation engine configured to dynamically assign an additional job process to the first computing system node based on determining that the utilization of at least one system resource of the first computing system node is below a threshold level, the determining based on the data representing utilization of system resources transmitted from the agent controller to the agent controller interface,

wherein the at least one system resource of the first computing system node is a first electronic random access memory capacity,

wherein the system resource allocation engine is configured to monitor a second electronic random access memory capacity of a second computing system node,

wherein the dynamically assigning by the system resource allocation engine of the additional job process comprises assigning the additional job process to the first computing system node based on determining that utilization of the first electronic random access memory capacity is below the threshold level,

wherein the system resource allocation engine is configured to prevent assignment of additional job processes to the second computing system node based on determining that utilization of the second electronic random access memory capacity is at or above a threshold value,

wherein the additional job process requires utilization of the first electronic random access memory capacity that is unused on the first computing system node.

11. The supervisor controller of claim 10, wherein the supervisor controller operates in a Hadoop computer cluster.

12. The supervisor controller of claim 11, wherein the management controller comprises a job tracker or a yarn resource manager, wherein the plurality of computer system nodes comprise task trackers or yarn node managers configured to communicate data to the job tracker or the yarn resource manager, the communicated data comprising a number of available slots or containers for processing tasks on a particular computer system node.

13. The supervisor controller of claim 12, wherein the supervisor controller is configured to assign the additional job process to the first computing system node based on determining that the utilization of the at least one system resource of the computing system node is below the threshold level and based on determining that the number of available slots or containers is zero for the computing system node.

14. The supervisor controller of claim 10, wherein the supervisor controller is configured to assign the additional job process to the first computing system node based on determining that the utilization of the at least one system resource of the first computing system node is below the threshold level and based on determining that the utilization of a second system resource of the first computing system node is above a second threshold level, the determining based on the monitored system resources utilization transmitted from the reporting controller to the supervisor controller.

15. The supervisor controller of claim 14, wherein the additional job process assigned by the supervisor controller requires greater utilization of the at least one system resource as compared to the utilization of the second system resource.

16. The supervisor controller of claim 10, wherein the plurality of computer system nodes comprises computing devices having heterogeneous machine types.

17. The supervisor controller of claim 16, wherein the first electronic random access memory capacity is larger than the second electronic random access memory capacity, wherein the assigning of the additional job process comprises assigning the additional job process to the first computing system based on determining that the utilization of the first electronic random access memory capacity is below the threshold level.

18. The supervisor controller claim 17, wherein the first computing system node is processing more job processes than the second computing system node.

19. A computer cluster comprising:

a management computing device comprising a supervisor controller configured to coordinate processing of a plurality of sub-jobs for a plurality of overall jobs;

a plurality of computer system nodes configured to communicate with the management computing device, and to perform processing of received sub-jobs, the computing system nodes each comprising;

one or more processors configured to perform computing processes on received sub-jobs;

an agent controller comprising;

a monitoring interface configured to monitor utilization by sub-jobs of system resources of a first computing system node; and

a reporting controller configured to transmit the monitored system resources utilization to the supervisor controller in substantially real-time;

wherein the supervisor controller is configured to assign an additional sub-job to the first computing system node based on determining that the utilization of at least one system resource of the first computing system node is below a threshold level, the determining based on the monitored system resources utilization transmitted from the reporting controller to the supervisor controller;

wherein the at least one system resource of the first computing system node is a first CPU processor capacity,

wherein the supervisor controller is configured to monitor a second CPU processor capacity of a second computing system node,

wherein the assigning by the supervisor controller of the additional sub-job comprises assigning the additional sub-job to the first computing system node based on determining that utilization of the first CPU processor capacity is below the threshold level,

wherein the supervisor controller is configured to prevent assignment of additional sub-jobs to the second computing system node based on determining that utilization of the second CPU processor capacity is at or above a threshold value,

wherein the additional sub-job requires utilization of the first CPU processor capacity that is unused on the first computing system node.

20. The computer cluster of claim 19, wherein the computer cluster is a Hadoop computer cluster.

21. The computer cluster of claim 20, wherein the management computing device comprises a management controller, the management controller comprises a job tracker or a yarn resource manager, wherein the plurality of computer system nodes comprise task trackers or yarn node managers configured to communicate data to the job tracker or the yarn resource manager, the communicated data comprising a number of available slots or containers for processing tasks on a particular computer system node.

22. The computer cluster of claim 21, wherein the supervisor controller is configured to assign the additional sub-job to the first computing system node based on determining that the utilization of the at least one system resource of the computing system node is below the threshold level and based on determining that the number of available slots or containers is zero for the first computing system node.

23. The computer cluster of claim 19, wherein the supervisor controller is configured to assign the additional sub-job to the first computing system node based on determining that the utilization of the at least one system resource of the first computing system node is below the threshold level and based on determining that the utilization of a second system resource of the first computing system node is below a second threshold level, the determining based on the monitored system resources utilization transmitted from the reporting controller to the supervisor controller.

24. The computer cluster of claim 23, wherein the additional sub-job assigned by the supervisor controller requires greater utilization of the at least one system resource as compared to the utilization of the second system resource.

25. The computer cluster of claim 19, wherein the plurality of computer system nodes comprises computing devices having heterogeneous machine types.

26. The computer cluster of claim 25, wherein the first CPU processor capacity is larger than the second CPU processor capacity, wherein the assigning of the additional sub-job comprises assigning the additional sub-job to the first computing system based on determining that the utilization of the first CPU processor capacity is below the threshold level.

27. The computer cluster claim 26, wherein the first computing system node is processing more sub-jobs than the second computing system node.

28. A supervisor controller configured to dynamically manage assignment of job processes in a computer cluster, the supervisor controller comprising:

a management controller interface configured to communicate with a management controller to access data representing an assignment of a plurality of job processes across a plurality of computer system nodes in the computer cluster;

an agent controller interface configured to communicate with an agent controller operating on a first computing system node, the agent controller configured to transmit to the agent controller interface data representing utilization of system resources by the plurality of job processes operating on the first computing system node; and

a system resource allocation engine configured to dynamically assign an additional job process to the first computing system node based on determining that the utilization of at least one system resource of the first computing system node is below a threshold level, the determining based on the data representing utilization of system resources transmitted from the agent controller to the agent controller interface,

wherein the at least one system resource of the first computing system node is a first CPU processor capacity,

wherein the system resource allocation engine is configured to monitor a second CPU processor capacity of a second computing system node,

wherein the dynamically assigning by the system resource allocation engine of the additional job process comprises assigning the additional job process to the first computing system node based on determining that utilization of the first CPU processor capacity is below the threshold level,

wherein the system resource allocation engine is configured to prevent assignment of additional job processes to the second computing system node based on determining that utilization of the second CPU processor capacity is at or above a threshold value,

wherein the additional job process requires utilization of the first CPU processor capacity that is unused on the first computing system node.

29. The supervisor controller of claim 28, wherein the supervisor controller operates in a Hadoop computer cluster.

30. The supervisor controller of claim 29, wherein the management controller comprises a job tracker or a yarn resource manager, wherein the plurality of computer system nodes comprise task trackers or yarn node managers configured to communicate data to the job tracker or the yarn resource manager, the communicated data comprising a number of available slots or containers for processing tasks on a particular computer system node.

31. The supervisor controller of claim 30, wherein the supervisor controller is configured to assign the additional job process to the first computing system node based on determining that the utilization of the at least one system resource of the computing system node is below the threshold level and based on determining that the number of available slots or containers is zero for the computing system node.

32. The supervisor controller of claim 28, wherein the supervisor controller is configured to assign the additional job process to the first computing system node based on determining that the utilization of the at least one system resource of the first computing system node is below the threshold level and based on determining that the utilization of a second system resource of the first computing system node is below a second threshold level, the determining based on the monitored system resources utilization transmitted from the reporting controller to the supervisor controller.

33. The supervisor controller of claim 32, wherein the additional job process assigned by the supervisor controller requires greater utilization of the at least one system resource as compared to the utilization of the second system resource.

34. The supervisor controller of claim 28, wherein the plurality of computer system nodes comprises computing devices having heterogeneous machine types.

35. The supervisor controller of claim 34, wherein the first CPU processor capacity is larger than the second CPU processor capacity, wherein the assigning of the additional job process comprises assigning the additional job process to the first computing system based on determining that the utilization of the first CPU processor capacity is below the threshold level.

36. The supervisor controller claim 35, wherein the first computing system node is processing more job processes than the second computing system node.