ADAPTIVE RESOURCE SCHEDULING FOR DATA STREAM PROCESSING

• US 20180270164A1
• Filed: 03/14/2017
• Published: 09/20/2018
• Est. Priority Date: 03/14/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: AHMED, KHALID, BAO, XIAO MING, FENG, KUAN, LIU, JUN FENG
• Granted Patent: US 10,554,577 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 9/38 Concurrent instruction execution, e.g. pipeline or look ahead
• CPC Class Codes: G06F 9/4806 Task transfer initiation or dispatching
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: G06F 9/5022 Mechanisms to release resources
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals
• CPC Class Codes: G06F 9/5038 considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration scheduling strategies G06F9/4881 and subgroups
• CPC Class Codes: H04L 47/801 Real time traffic
• CPC Class Codes: H04L 47/822 Collecting or measuring resource availability data
• CPC Class Codes: H04L 47/823 Prediction of resource usage
• CPC Class Codes: H04L 47/83 based on usage prediction
• CPC Class Codes: H04L 67/62 Establishing a time schedule for servicing the requests

Abstract

In at least one embodiment of stream processing in a data processing system cluster, a driver requests, from a cluster manager, execution of a first task within a job including a plurality of tasks, as well as a first quantity of resources of the cluster to support execution of the task. At completion of execution of the first task by the cluster, the driver retains for a second task among the plurality of tasks at least a portion of the resources allocated to the first task by the cluster manager. The driver determines a second quantity of the resources to retain for the second task based on a relative priority of the job to other jobs in the cluster and efficiency of the cluster manager in allocating resources to the first task.

Claims

1. A method of stream processing in a data processing system cluster, the method comprising:

a processor executing a driver to request, from a cluster manager, (1) execution of a first task within a job including a plurality of tasks and (2) a first quantity of resources of the cluster to support execution of the task; and

at completion of execution of the first task by the cluster, the processor executing the driver to retain for a second task among the plurality of tasks at least a portion of the resources allocated to the first task by the cluster manager, wherein the retaining includes;

determining a second quantity of the resources to retain for the second task based on a relative priority of the job to other jobs in the cluster and efficiency of the cluster manager in allocating resources to the first task.

2. The method of claim 1, wherein the processor executing the driver to retain for the second task at least a portion of the resources allocated to the first task includes determining to retain the at least a portion of the resources allocated to the first task by the cluster manager only in response to determining a resource allocation time is less than a resource reuse time.

3. The method of claim 1, wherein the processor executing the driver to request, from the cluster manager, execution of the first task and the first quantity of resources includes determining the first quantity of resources based on a resource allocation time and a task execution time.

4. The method of claim 3, wherein the first quantity of resources is further based on a third quantity of resources retained following execution of a third task among the plurality of tasks that precedes the first task.

5. The method of claim 3, wherein the resource allocation time is a weighted resource allocation time derived across multiple tasks.

6. The method of claim 1, and further comprising the processor executing the driver to request, from the cluster manager, execution of the second task within the job.

7. A data processing system, comprising:

a processor;

a network interface configured to couple the data processing system to a cluster of data processing systems;

data storage coupled to the processor, wherein the data storage includes driver program code that, when executed by the processor, causes the data processing system to perform;

requesting, from a cluster manager, (1) execution of a first task within a job including a plurality of tasks and (2) a first quantity of resources of the cluster to support execution of the task; and

at completion of execution of the first task by the cluster, retaining for a second task among the plurality of tasks at least a portion of the resources allocated to the first task by the cluster manager, wherein the retaining includes;

determining a second quantity of the resources to retain for the second task based on a relative priority of the job to other jobs in the cluster and efficiency of the cluster manager in allocating resources to the first task.

8. The data processing system of claim 7, wherein the retaining includes determining to retain the at least a portion of the resources allocated to the first task by the cluster manager only in response to determining a resource allocation time is less than a resource reuse time.

9. The data processing system of claim 7, wherein the requesting includes determining the first quantity of resources based on a resource allocation time and a task execution time.

10. The data processing system of claim 9, wherein the first quantity of resources is further based on a third quantity of resources retained following execution of a third task among the plurality of tasks that precedes the first task.

11. The data processing system of claim 9, wherein the resource allocation time is a weighted resource allocation time derived across multiple tasks.

12. The data processing system of claim 7, wherein the driver program code, when executed by the processor, further causes the data processing system to perform:

requesting, from the cluster manager, execution of the second task within the job.

14. The data processing system of claim 7, wherein the retaining includes determining to retain the at least a portion of the resources allocated to the first task by the cluster manager only in response to determining a resource allocation time is less than a resource reuse time.

15. The data processing system of claim 7, wherein the requesting includes determining the first quantity of resources based on a resource allocation time and a task execution time.

16. The data processing system of claim 9, wherein the first quantity of resources is further based on a third quantity of resources retained following execution of a third task among the plurality of tasks that precedes the first task.

17. The data processing system of claim 9, wherein the resource allocation time is a weighted resource allocation time derived across multiple tasks.

18. The data processing system of claim 7, wherein the program instructions, when executed by the data processing system, further cause the data processing system to perform:

requesting, from the cluster manager, execution of the second task within the job.

13. A computer program product, the computer program product comprising a computer readable storage device having program instructions embodied therewith, the program instructions executable by a data processing system, to cause the data processing system to perform:

requesting, from a cluster manager, (1) execution of a first task within a job including a plurality of tasks and (2) a first quantity of resources of the cluster to support execution of the task; and

at completion of execution of the first task by the cluster, retaining for a second task among the plurality of tasks at least a portion of the resources allocated to the first task by the cluster manager, wherein the retaining includes;

determining a second quantity of the resources to retain for the second task based on a relative priority of the job to other jobs in the cluster and efficiency of the cluster manager in allocating resources to the first task.