Adaptive datacenter topology for distributed frameworks job control through network awareness

• US 9,785,522 B2
• Filed: 11/14/2014
• Issued: 10/10/2017
• Est. Priority Date: 11/14/2014
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Kulkarni, Karthik Krishna, Nambiar, Raghunath
• Primary Examiner(s): Tang, Kenneth
• Application Number:
  US14/541,976
• Publication Number:
  US 20160140001A1
• Time in Patent Office: 1,061 Days
• Field of Search: None
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3024 where the computing system component is a central processing unit [CPU]
• CPC Class Codes: G06F 11/3404 for parallel or distributed programming
• CPC Class Codes: G06F 11/3428 Benchmarking
• CPC Class Codes: G06F 11/3433 for load management allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2209/5011 Pool
• CPC Class Codes: G06F 2209/5021 Priority
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451
• CPC Class Codes: H04L 41/147 for predicting network behaviour
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

Systems, methods, and computer program products to perform an operation comprising receiving a priority of a distributed computing job, an intermediate traffic type of the distributed computing job, and a set of candidate compute nodes available to process the distributed computing job, the candidate compute nodes each available to process at least one input split of the distributed computing job, and selecting a mapper node from the candidate compute nodes, for one of the input splits, wherein the mapper node is selected based on the priority and the intermediate traffic type of the distributed computing job, wherein the mapper compute node is further selected upon determining that the mapper node is not affected by an error, and a resource utilization score for the mapper node does not exceed a utilization threshold.

Claims

1. A method, comprising:

receiving a priority of a distributed computing job and a set of candidate compute nodes available to process the distributed computing job, the candidate compute nodes each available to process at least one input split, of a plurality of input splits, wherein the plurality of input splits comprise a data input of the distributed computing job;

determining, based at least in part on a source code of the distributed computing job, that an intermediate traffic type of the distributed computing job is a first intermediate traffic type, of a plurality of intermediate traffic types, wherein the intermediate traffic type reflects a size of a data output generated by processing the distributed computing job relative to a size of the data input of the distributed computing job; and

selecting a first compute node of the candidate compute nodes as a mapper node for one of the input splits, wherein the first compute node is selected based on the priority and the intermediate traffic type of the distributed computing job and an overall score computed for the first candidate compute node, wherein the first compute node is further selected upon determining that;

the first compute node is not experiencing an error; and

a resource utilization score for the first compute node does not exceed a utilization threshold.

2. The method of claim 1, wherein the priority of the distributed computing job comprises a high priority job relative to other distributed computing jobs, wherein the first compute node is selected upon determining that the overall score for the first compute node exceeds an overall score of each other candidate compute node,

wherein the intermediate traffic type of the distributed computing type is further determined based on:

(i) an intermediate traffic type of each of a plurality of previous executions of the distributed computing job, and (ii) an intermediate traffic type of each of a plurality of previous executions of other distributed computing jobs, the method further comprising;

computing the overall score for each of the plurality of candidate compute nodes based on;

(i) a resource utilization score of the respective candidate compute node, (ii) a number of errors affecting the respective candidate compute node, (iii) a type of the errors affecting the respective candidate compute node, (iv) a hardware configuration of the respective candidate compute node, (v) a current level of utilization of system resources by the respective candidate compute node, and (vi) a forecasted level of utilization of the system resources by the respective candidate compute node; and

removing a second compute node from the set of candidate compute nodes upon determining the intermediate traffic type of the distributed computing job is the first intermediate traffic type, wherein the second compute node does not process any of the input splits of the distributed computing job.

3. The method of claim 1, wherein the error comprises at least one of a network error and an error of the first compute node, the method further comprising:

upon determining that the first compute node is experiencing the error;

removing the first compute node from the set of candidate compute nodes; and

selecting a second compute node, of the candidate compute nodes, for the first input split, wherein the second compute node is selected based on the priority and the intermediate traffic type of the distributed computing job, wherein the second compute node is further selected upon determining that;

the second compute node is not experiencing an error; and

a resource utilization score for the second compute node does not exceed the utilization threshold.

4. The method of claim 1, wherein the resource utilization score of the first compute node is less than the utilization threshold and greater than a minimum utilization threshold, wherein the first compute node is selected as the mapper node for the first input split upon further determining:

(i) that each other candidate compute node is experiencing an error, and (ii) that the resource utilization score of each of the other candidate compute nodes exceeds the utilization threshold.

5. The method of claim 1, further comprising:

determining that a second compute node, of the plurality of compute nodes, is not experiencing an error;

determining that a resource utilization score of the second compute node does not exceed the utilization threshold; and

selecting the second node as a reduce node for the distributed computing job.

6. The method of claim 1, wherein the resource utilization score is based on:

(i) a current level of utilization of system resources by the first compute node, and (ii) a forecasted level of utilization of the system resources by the first compute node, wherein the system resources comprise;

(i) a processor (CPU), (ii) a network bandwidth, (iii) a memory, and (iv) a system input/output (I/O).

7. The method of claim 1, further comprising:

upon determining that a second input split for a different distributed computing job is being processed by the first compute node;

preempting the processing of the second input split by the first compute node; and

processing the first input split by the first compute node, wherein the distributed computing job has a greater priority than the different distributed computing job.

8. A system, comprising:

a computer processor; and

a memory containing a program, which when executed by the processor, performs an operation comprising;

receiving a priority of a distributed computing job and a set of candidate compute nodes available to process the distributed computing job, the candidate compute nodes each available to process at least one input split, of a plurality of input splits, wherein the plurality of input splits comprise a data input of the distributed computing job;

determining, based at least in part on a source code of the distributed computing job, that an intermediate traffic type of the distributed computing job is a first intermediate traffic type, of a plurality of intermediate traffic types, wherein the intermediate traffic type reflects a size of a data output generated by processing the distributed computing job relative to a size of the data input of the distributed computing job; and

selecting a first compute node of the candidate compute nodes as a mapper node for one of the input splits, wherein the first compute node is selected based on the priority and the intermediate traffic type of the distributed computing job and an overall score computed for the first candidate compute node, wherein the first compute node is further selected upon determining that;

the first compute node is not experiencing an error; and

a resource utilization score for the first compute node does not exceed a utilization threshold.

9. The system of claim 8, wherein the priority of the distributed computing job comprises a high priority job relative to other distributed computing jobs, wherein the first compute node is selected upon determining that an overall score for the first compute node exceeds an overall score of each other candidate compute node,

wherein the intermediate traffic type of the distributed computing type is further determined based on:

(i) an intermediate traffic type of each of a plurality of previous executions of the distributed computing job, and (ii) an intermediate traffic type of each of a plurality of previous executions of other distributed computing jobs, the operation further comprising;

computing the overall score for each of the plurality of candidate compute nodes based on;

(i) a resource utilization score of the respective candidate compute node, (ii) a number of errors affecting the respective candidate compute node, (iii) a type of the errors affecting the respective candidate compute node, (iv) a hardware configuration of the respective candidate compute node, (v) a current level of utilization of system resources by the respective candidate compute node, and (vi) a forecasted level of utilization of the system resources by the respective candidate compute node; and

removing a second compute node from the set of candidate compute nodes upon determining the intermediate traffic type of the distributed computing job is the first intermediate traffic type, wherein the second compute node does not process any of the input splits of the distributed computing job.

10. The system of claim 8, wherein the error comprises at least one of a network error and an error of the first compute node, the operation further comprising:

upon determining that the first compute node is experiencing the error;

removing the first compute node from the set of candidate compute nodes; and

selecting a second compute node, of the candidate compute nodes, for the first input split, wherein the second compute node is selected based on the priority and the intermediate traffic type of the distributed computing job, wherein the second compute node is further selected upon determining that;

the second compute node is not experiencing an error; and

a resource utilization score for the second compute node does not exceed the utilization threshold.

11. The system of claim 8, wherein the resource utilization score of the first compute node is less than the utilization threshold and greater than a minimum utilization threshold, wherein the first compute node is selected as the mapper node for the first input split upon further determining:

(i) that each other candidate compute node is experiencing an error, and (ii) that the resource utilization score of each of the other candidate compute nodes exceeds the utilization threshold.

12. The system of claim 8, the operation further comprising:

determining that a second compute node, of the plurality of compute nodes, is not experiencing an error;

determining that a resource utilization score of the second compute node does not exceed the utilization threshold; and

selecting the second node as a reduce node for the distributed computing job.

13. The system of claim 8, wherein the resource utilization score is based on:

(i) a current level of utilization of system resources by the first compute node, and (ii) a forecasted level of utilization of the system resources by the first compute node, wherein the system resources comprise;

(i) a processor (CPU), (ii) a network bandwidth, (iii) a memory, and (iv) a system input/output (I/O).

14. The system of claim 8, the operation further comprising:

upon determining that a second input split for a different distributed computing job is being processed by the first compute node;

preempting the processing of the second input split by the first compute node; and

processing the first input split by the first compute node, wherein the distributed computing job has a greater priority than the different distributed computing job.

15. A computer program product, comprising:

a non-transitory computer-readable storage medium having computer-readable code embodied therewith, the computer-readable program code executable by a processor to perform an operation to optimize placement of a distributed computing job on a plurality of compute nodes, the operation comprising;

receiving a priority of a distributed computing job and a set of candidate compute nodes available to process the distributed computing job, the candidate compute nodes each available to process at least one input split, of a plurality of input splits, wherein the plurality of input splits comprise a data input of the distributed computing job;

determining, based at least in part on a source code of the distributed computing job, that an intermediate traffic type of the distributed computing job is a first intermediate traffic type, of a plurality of intermediate traffic types, wherein the intermediate traffic type reflects a size of a data output generated by processing the distributed computing job relative to a size of the data input of the distributed computing job; and

selecting a first compute node of the candidate compute nodes as a mapper node for one of the input splits, wherein the first compute node is selected based on the priority and the intermediate traffic type of the distributed computing job and an overall score computed for the first candidate compute node, wherein the first compute node is further selected upon determining that;

the first compute node is not experiencing an error; and

a resource utilization score for the first compute node does not exceed a utilization threshold.

16. The computer program product of claim 15, wherein the priority of the distributed computing job comprises a high priority job relative to other distributed computing jobs, wherein the first compute node is selected upon determining that an overall score for the first compute node exceeds an overall score of each other candidate compute node,

wherein the intermediate traffic type of the distributed computing type is further determined based on:

(i) an intermediate traffic type of each of a plurality of previous executions of the distributed computing job, and (ii) an intermediate traffic type of each of a plurality of previous executions of other distributed computing jobs, the operation further comprising;

computing the overall score for each of the plurality of candidate compute nodes based on;

(i) a resource utilization score of the respective candidate compute node, (ii) a number of errors affecting the respective candidate compute node, (iii) a type of the errors affecting the respective candidate compute node, (iv) a hardware configuration of the respective candidate compute node, (v) a current level of utilization of system resources by the respective candidate compute node, and (vi) a forecasted level of utilization of the system resources by the respective candidate compute node; and

removing a second compute node from the set of candidate compute nodes upon determining the intermediate traffic type of the distributed computing job is the first intermediate traffic type, wherein the second compute node does not process any of the input splits of the distributed computing job.

17. The computer program product of claim 15, wherein the error comprises at least one of a network error and an error of the first compute node, the operation further comprising:

upon determining that the first compute node is experiencing the error;

removing the first compute node from the set of candidate compute nodes; and

selecting a second compute node, of the candidate compute nodes, for the first input split, wherein the second compute node is selected based on the priority and the intermediate traffic type of the distributed computing job, wherein the second compute node is further selected upon determining that;

the second compute node is not experiencing an error; and

a resource utilization score for the second compute node does not exceed the utilization threshold.

18. The computer program product of claim 15, wherein the resource utilization score of the first compute node is less than the utilization threshold and greater than a minimum utilization threshold, wherein the first compute node is selected as the mapper node for the first input split upon further determining:

(i) that each other candidate compute node is experiencing an error, and (ii) that the resource utilization score of each of the other candidate compute nodes exceeds the utilization threshold.

19. The computer program product of claim 15, the operation further comprising:

determining that a second compute node, of the plurality of compute nodes, is not experiencing an error;

determining that a resource utilization score of the second compute node does not exceed the utilization threshold; and

selecting the second node as a reduce node for the distributed computing job.

20. The computer program product of claim 15, wherein the resource utilization score is based on:

(i) a current level of utilization of system resources by the first compute node, and (ii) a forecasted level of utilization of the system resources by the first compute node, wherein the system resources comprise;

(i) a processor (CPU), (ii) a network bandwidth, (iii) a memory, and (iv) a system input/output (I/O).