Systems, methods, and devices for dynamic resource monitoring and allocation in a cluster system
DCFirst Claim
Patent Images
1. A hadoop computer cluster comprising:
- one or more processors of a master node, wherein the master node comprises a supervisor controller;
one or more processors of a plurality of computing system nodes, the one or more processors of the plurality of computing system nodes configured to perform computing processes on received sub-jobs, wherein each computing system node comprises an agent controller;
wherein each agent controller comprises;
a monitoring interface configured to monitor system resources utilization by sub-jobs of its respective computing system node; and
a reporting controller configured to transmit the monitored system resources utilization to the supervisor controller in substantially real-time;
wherein the supervisor controller is configured to assign an additional sub-job to a first computing system node based on determining that the utilization of a first electronic random access memory capacity of the first computing system node is below a threshold level, the determining based on the monitored system resources utilization transmitted from the reporting controller of the first computing system node to the supervisor controller,wherein the supervisor controller is configured to monitor a second electronic random access memory capacity of a second computing system node,wherein the supervisor controller is configured to prevent assignment of additional sub-jobs to a second computing system node based on determining that utilization of the second electronic random access memory capacity is at or above a threshold value, andwherein the master node and the plurality of computing system nodes include a computer processor and an electronic storage medium.
2 Assignments
Litigations
0 Petitions
Accused Products
Abstract
In an embodiment, the systems, methods, and devices disclosed herein comprise a computer resource monitoring and allocation system. In an embodiment, the resource monitoring and allocation system can be configured to allocate computer resources that are available on various nodes of a cluster to specific jobs and/or sub-jobs and/or tasks and/or processes.
-
Citations
20 Claims
-
1. A hadoop computer cluster comprising:
-
one or more processors of a master node, wherein the master node comprises a supervisor controller; one or more processors of a plurality of computing system nodes, the one or more processors of the plurality of computing system nodes configured to perform computing processes on received sub-jobs, wherein each computing system node comprises an agent controller; wherein each agent controller comprises; a monitoring interface configured to monitor system resources utilization by sub-jobs of its respective computing system node; and a reporting controller configured to transmit the monitored system resources utilization to the supervisor controller in substantially real-time; wherein the supervisor controller is configured to assign an additional sub-job to a first computing system node based on determining that the utilization of a first electronic random access memory capacity of the first computing system node is below a threshold level, the determining based on the monitored system resources utilization transmitted from the reporting controller of the first computing system node to the supervisor controller, wherein the supervisor controller is configured to monitor a second electronic random access memory capacity of a second computing system node, wherein the supervisor controller is configured to prevent assignment of additional sub-jobs to a second computing system node based on determining that utilization of the second electronic random access memory capacity is at or above a threshold value, and wherein the master node and the plurality of computing system nodes include a computer processor and an electronic storage medium. - View Dependent Claims (2, 3, 4, 5)
-
-
6. A supervisor controller configured to dynamically manage assignment of job processes in a hadoop computer cluster, the supervisor controller comprising:
-
a management controller interface configured to communicate with a management controller to access data representing an assignment of a plurality of job processes across a plurality of computer system nodes in the computer cluster; an agent controller interface configured to communicate with an agent controller operating on a first computing system node to receive data representing utilization of system resources on the first computing system node, wherein the agent controller interface is further configured to communicate with an agent controller operating on a second computing system node to receive data representing utilization of system resources on the second computing system node; and a system resource allocation engine configured to assign an additional job process to the first computing system node based on determining that the utilization of a first electronic random access memory capacity of the first computing system node is below a threshold level; and wherein one or more computer processors and one or more electronic storage medium are configured to operate the supervisor controller; wherein the system resource allocation engine is configured to monitor a second electronic random access memory capacity of a second computing system node, and wherein the system resource allocation engine is configured to prevent assignment of additional job processes to a second computing system node based on determining that utilization of the second electronic random access memory capacity is at or above a threshold level. - View Dependent Claims (7, 8, 9, 10)
-
-
11. A hadoop computer cluster comprising:
-
one or more processors of a master node, wherein the master node comprises a supervisor controller; one or more processors of a plurality of computing system nodes, the one or more processors of the plurality of computing system nodes configured to perform computing processes on received sub-jobs, wherein each computing system node comprises an agent controller; wherein each agent controller comprises; a monitoring interface configured to monitor system resources utilization by sub-jobs of its respective computing system node; and a reporting controller configured to transmit the monitored system resources utilization to the supervisor controller in substantially real-time; wherein the supervisor controller is configured to assign an additional sub-job to a first computing system node based on determining that the utilization of a first CPU processor capacity of the first computing system node is below a threshold level, the determining based on the monitored system resources utilization transmitted from the reporting controller of the first computing system node to the supervisor controller; wherein the supervisor controller is configured to monitor a second CPU processor capacity of a second computing system node, wherein the supervisor controller is configured to prevent assignment of additional sub-jobs to the second computing system node based on determining that utilization of the second CPU processor capacity is at or above a threshold value, and wherein the master node and the plurality of computing system nodes include a computer processor and an electronic storage medium. - View Dependent Claims (12, 13, 14, 15)
-
-
16. A supervisor controller configured to manage assignment of job processes in a computer cluster, the supervisor controller comprising:
-
a management controller interface configured to communicate with a management controller to access data representing an assignment of a plurality of job processes across a plurality of computing system nodes in the computer cluster; an agent controller interface configured to communicate with an agent controller operating on a first computing system node to receive data representing utilization of system resources by the plurality of job processes operating on the first computing system node, wherein the agent controller interface is further configured to communicate with an agent controller operating on a second computing system node to receive data representing utilization of system resources on the second computing system node; and a system resource allocation engine configured to assign an additional job process to a first computing system node based on determining that the utilization of a first CPU processor capacity of the first computing system node is below a threshold level; and wherein one or more computer processors and one or more electronic storage medium are configured to operate the supervisor controller; wherein the system resource allocation engine is configured to monitor a second CPU processor capacity of a second computing system node, and wherein the system resource allocation engine is configured to prevent assignment of additional job processes to the second computing system node based on determining that utilization of the second CPU processor capacity is at or above a threshold value. - View Dependent Claims (17, 18, 19, 20)
-
Specification