Techniques for dynamically assigning jobs to processors in a cluster based on inter-thread communications
First Claim
Patent Images
1. A method of operating a high performance computing cluster that includes multiple nodes that each include multiple processors and monitoring hardware, the method comprising:
- monitoring, by the monitoring hardware, communication between a plurality of threads assigned to the multiple processors;
periodically broadcasting, by the monitoring hardware, information, related to a level of processor utilization and network utilization at each of the multiple nodes, from each of the multiple nodes to remaining ones of the multiple nodes;
updating, by the monitoring hardware, respective local job tables maintained in each of the multiple nodes based on the broadcast information; and
reassigning, by the monitoring hardware, based on the broadcast information in the respective local job tables, at least one of the one or more threads to a different one of the multiple processors such that threads of a job complete at substantially the same time, wherein the reassigning includes;
cracking one of the threads that is executing on a first processor, included in the multiple processors, into at least two secondary threads based on a first workload of the first processor;
moving one of the at least two secondary threads to a second processor, included in the multiple processors, based on a second workload of the second processor; and
moving identified threads of the plurality of threads that communicate above a threshold level to identified processors of the multiple processors that are located physically closer to each other than processors of the multiple processors that previously executed the identified threads.
2 Assignments
0 Petitions
Accused Products
Abstract
A technique for operating a high performance computing (HPC) cluster includes monitoring communication between threads assigned to multiple processors included in the HPC cluster. The HPC cluster includes multiple nodes that each include two or more of the multiple processors. One or more of the threads are moved to a different one of the multiple processors based on the communication between the threads.
-
Citations
14 Claims
-
1. A method of operating a high performance computing cluster that includes multiple nodes that each include multiple processors and monitoring hardware, the method comprising:
-
monitoring, by the monitoring hardware, communication between a plurality of threads assigned to the multiple processors; periodically broadcasting, by the monitoring hardware, information, related to a level of processor utilization and network utilization at each of the multiple nodes, from each of the multiple nodes to remaining ones of the multiple nodes; updating, by the monitoring hardware, respective local job tables maintained in each of the multiple nodes based on the broadcast information; and reassigning, by the monitoring hardware, based on the broadcast information in the respective local job tables, at least one of the one or more threads to a different one of the multiple processors such that threads of a job complete at substantially the same time, wherein the reassigning includes; cracking one of the threads that is executing on a first processor, included in the multiple processors, into at least two secondary threads based on a first workload of the first processor; moving one of the at least two secondary threads to a second processor, included in the multiple processors, based on a second workload of the second processor; and moving identified threads of the plurality of threads that communicate above a threshold level to identified processors of the multiple processors that are located physically closer to each other than processors of the multiple processors that previously executed the identified threads. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A high performance computing cluster, comprising:
-
multiple nodes that each include multiple processors; and monitoring hardware included in each of the multiple nodes, wherein the monitoring hardware is configured to monitor communication between a plurality of threads assigned to the multiple processors, periodically broadcast information, related to a level of processor utilization and network utilization at each of the multiple nodes, from each of the multiple nodes to remaining ones of the multiple nodes, update respective local job tables maintained in each of the multiple nodes based on the broadcast information, and move, based on the broadcast information in the respective local job tables, at least one of the one or more threads to a different one of the multiple processors such that threads of a job complete at substantially the same time, and wherein the move includes cracking one of the threads that is executing on a first processor, included in the multiple processors, into at least two secondary threads based on a first workload of the first processor, moving one of the at least two secondary threads to a second processor, included in the multiple processors, based on a second workload of the second processor, and moving identified threads of the plurality of threads that communicate above a threshold level to identified processors of the multiple processors that are located physically closer to each other than processors of the multiple processors that previously executed the identified threads. - View Dependent Claims (9, 10, 11, 12, 13)
-
-
14. A method of operating a high performance computing cluster that includes multiple nodes that each include multiple processors and monitoring hardware, the method comprising:
-
monitoring, by the monitoring hardware, communication between a plurality of threads assigned to the multiple processors periodically broadcasting, by the monitoring hardware, information, related to a level of processor utilization and network utilization at each of the multiple nodes, from each of the multiple nodes to remaining ones of the multiple nodes; updating, by the monitoring hardware, respective local job tables maintained in each of the multiple nodes based on the broadcast information; and moving, by the monitoring hardware, based on the broadcast information in the respective local job tables, at least one of the one or more threads to a different one of the multiple processors such that threads of a job complete at substantially the same time, wherein the information is broadcast using a message passing interface and the multiple nodes are arranged in a three-dimensional Torus topology, and wherein the moving includes; cracking one of the threads that is executing on a first processor, included in the multiple processors, into at least two secondary threads based on a first workload of the first processor; moving one of the at least two secondary threads to a second processor, included in the multiple processors, based on a second workload of the second processor; and moving identified threads of the plurality of threads that communicate above a threshold level to identified processors of the multiple processors that are located physically closer to each other than processors of the multiple processors that previously executed the identified threads.
-
Specification