ADAPTIVE RESOURCE SCHEDULING FOR DATA STREAM PROCESSING
First Claim
1. A method of stream processing in a data processing system cluster, the method comprising:
- a processor executing a driver to request, from a cluster manager, (1) execution of a first task within a job including a plurality of tasks and (2) a first quantity of resources of the cluster to support execution of the task; and
at completion of execution of the first task by the cluster, the processor executing the driver to retain for a second task among the plurality of tasks at least a portion of the resources allocated to the first task by the cluster manager, wherein the retaining includes;
determining a second quantity of the resources to retain for the second task based on a relative priority of the job to other jobs in the cluster and efficiency of the cluster manager in allocating resources to the first task.
1 Assignment
0 Petitions
Accused Products
Abstract
In at least one embodiment of stream processing in a data processing system cluster, a driver requests, from a cluster manager, execution of a first task within a job including a plurality of tasks, as well as a first quantity of resources of the cluster to support execution of the task. At completion of execution of the first task by the cluster, the driver retains for a second task among the plurality of tasks at least a portion of the resources allocated to the first task by the cluster manager. The driver determines a second quantity of the resources to retain for the second task based on a relative priority of the job to other jobs in the cluster and efficiency of the cluster manager in allocating resources to the first task.
27 Citations
18 Claims
-
1. A method of stream processing in a data processing system cluster, the method comprising:
-
a processor executing a driver to request, from a cluster manager, (1) execution of a first task within a job including a plurality of tasks and (2) a first quantity of resources of the cluster to support execution of the task; and at completion of execution of the first task by the cluster, the processor executing the driver to retain for a second task among the plurality of tasks at least a portion of the resources allocated to the first task by the cluster manager, wherein the retaining includes; determining a second quantity of the resources to retain for the second task based on a relative priority of the job to other jobs in the cluster and efficiency of the cluster manager in allocating resources to the first task. - View Dependent Claims (2, 3, 4, 5, 6)
-
-
7. A data processing system, comprising:
-
a processor; a network interface configured to couple the data processing system to a cluster of data processing systems; data storage coupled to the processor, wherein the data storage includes driver program code that, when executed by the processor, causes the data processing system to perform; requesting, from a cluster manager, (1) execution of a first task within a job including a plurality of tasks and (2) a first quantity of resources of the cluster to support execution of the task; and at completion of execution of the first task by the cluster, retaining for a second task among the plurality of tasks at least a portion of the resources allocated to the first task by the cluster manager, wherein the retaining includes; determining a second quantity of the resources to retain for the second task based on a relative priority of the job to other jobs in the cluster and efficiency of the cluster manager in allocating resources to the first task. - View Dependent Claims (8, 9, 10, 11, 12, 14, 15, 16, 17, 18)
-
-
13. A computer program product, the computer program product comprising a computer readable storage device having program instructions embodied therewith, the program instructions executable by a data processing system, to cause the data processing system to perform:
-
requesting, from a cluster manager, (1) execution of a first task within a job including a plurality of tasks and (2) a first quantity of resources of the cluster to support execution of the task; and at completion of execution of the first task by the cluster, retaining for a second task among the plurality of tasks at least a portion of the resources allocated to the first task by the cluster manager, wherein the retaining includes; determining a second quantity of the resources to retain for the second task based on a relative priority of the job to other jobs in the cluster and efficiency of the cluster manager in allocating resources to the first task.
-
Specification