AUTOMATED CLOUD WORKLOAD MANAGEMENT IN A MAP-REDUCE ENVIRONMENT

• US 20110154350A1
• Filed: 12/18/2009
• Published: 06/23/2011
• Est. Priority Date: 12/18/2009
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Kaminsky, David L., Doyle, Ronald P.
• Granted Patent: US 9,141,433 B2
• Time in Patent Office: Days
• US Class Current: 718/104
• CPC Class Codes: G06F 9/5044 considering hardware capabilities
• CPC Class Codes: G06F 9/5072 Grid computing
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/61 taking into account QoS or priority requirements
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A computing device associated with a cloud computing environment identifies a first worker cloud computing device from a group of worker cloud computing devices with available resources sufficient to meet required resources for a highest-priority task associated with a computing job including a group of prioritized tasks. A determination is made as to whether an ownership conflict would result from an assignment of the highest-priority task to the first worker cloud computing device based upon ownership information associated with the computing job and ownership information associated with at least one other task assigned to the first worker cloud computing device. The highest-priority task is assigned to the first worker cloud computing device in response to determining that the ownership conflict would not result from the assignment of the highest-priority task to the first worker cloud computing device.

Claims

1. A method, comprising:

identifying, via a computing device associated with a cloud computing environment, a first worker cloud computing device from a plurality of worker cloud computing devices with available resources sufficient to meet required resources for a highest-priority task associated with a computing job comprising a plurality of prioritized tasks;

determining, based upon ownership information associated with the computing job and ownership information associated with at least one other task assigned to the first worker cloud computing device, whether an ownership conflict would result from an assignment of the highest-priority task to the first worker cloud computing device; and

assigning the highest-priority task to the first worker cloud computing device in response to determining that the ownership conflict would not result from the assignment of the highest-priority task to the first worker cloud computing device.

2. The method of claim 1, further comprising:

estimating computing resources required to complete each of the plurality of prioritized tasks; and

identifying the highest-priority task to assign to the first worker cloud computing device based upon priority information associated with each of the prioritized tasks and the computing resources required to complete each of the plurality of prioritized tasks.

3. The method of claim 1, further comprising:

determining, based upon the ownership information associated with the computing job and the ownership information associated with the at least one other task assigned to the first worker cloud computing device, that the ownership conflict would result from the assignment of the highest-priority task to the first worker cloud computing device;

identifying a second worker cloud computing device with available resources sufficient to meet the required resources for the highest-priority task without causing the ownership conflict based upon the ownership information associated with the computing job and ownership information associated with at least one other task assigned to the second worker cloud computing device; and

assigning the highest-priority task to the second worker cloud computing device in response to determining that the second worker cloud computing device comprises the available resources sufficient to meet the required resources for the highest-priority task without causing the ownership conflict.

4. The method of claim 1, where identifying, via the computing device associated with the cloud computing environment, the first worker cloud computing device from the plurality of worker cloud computing devices with the available resources sufficient to meet the required resources for the highest-priority task associated with the computing job comprises:

querying each of the plurality of worker cloud computing devices for available resources of that worker cloud computing device; and

receiving a response from each of the plurality of worker cloud computing devices with the available resources of that worker cloud computing device.

5. The method of claim 4, where the available resources of each of the plurality of worker cloud computing devices comprise one of available processor resources and available input/output (I/O) resources, and further comprising:

determining which of processor resource requirements and I/O resource requirements for the highest-priority task is greater; and

identifying the first worker cloud computing device among the plurality of worker cloud computing devices with a greatest of the one of the available processor resources and the available input/output (I/O) resources that match the determined greater of the processor resource requirements and I/O resource requirements for the highest-priority task.

6. The method of claim 1, further comprising:

receiving updated available resources from the first worker cloud computing device in response to assigning the highest-priority task to the first worker cloud computing device;

determining which of the first worker cloud computing device and other of the plurality of worker cloud computing devices has a greatest resource availability to meet required resources for a next highest-priority task associated with the computing job;

determining, based upon the ownership information associated with the computing job and ownership information associated with at least one other task assigned to the worker cloud computing device determined to have the greatest available resources, whether an additional ownership conflict would result from an assignment of the next highest-priority task to that worker cloud computing device; and

assigning the next highest-priority task to the worker cloud computing device determined to have the greatest resource availability to meet the required resources for the next highest-priority task in response to determining that the additional ownership conflict would not result from the assignment of the next highest-priority task to that worker cloud computing device.

7. The method of claim 1, further comprising:

iteratively assigning remaining tasks associated with the computing job to additional ones of the plurality of worker cloud computing devices in decreasing order of priority and based upon highest available resources among the additional ones of the plurality of worker cloud computing devices for each iteration of task assignment without causing additional ownership conflicts.

8. A method, comprising:

receiving, at a worker cloud computing device within a cloud computing environment, an indication of an available computing job comprising ownership information and priority information;

determining available resources of the worker cloud computing device;

determining ownership information for a task associated with another computing job presently being processed by the worker cloud computing device; and

selecting a highest-priority task associated with the computing job for execution in response to determining that the available resources of the worker cloud computing device are sufficient to meet required resources for the highest-priority task without causing an ownership conflict between the computing job and the other computing job presently being processed by the worker cloud computing device.

9. The method of claim 8, where selecting the highest-priority task associated with the computing job for execution comprises selecting the highest-priority task from a centralized job database comprising a plurality of partitioned prioritized tasks associated with the computing job, where the partitioned prioritized tasks are prioritized based upon the priority information.

10. The method of claim 8, further comprising:

determining updated available resources in response to beginning execution of the highest-priority task; and

selecting a next highest-priority task associated with the computing job for execution upon determining that the updated available resources of the worker computing device are sufficient to meet required resources for the next highest-priority task without causing the ownership conflict.

11. The method of claim 8, further comprising:

determining that the task associated with the other computing job presently being processed by the worker cloud computing device comprises a priority lower than the highest-priority task;

pausing the lower-priority task presently being processed by the worker cloud computing device; and

resuming execution of the lower-priority task in response to determining that execution of the highest-priority task has completed.

12. A system, comprising:

a memory; and

a processor programmed to;

retrieve a computing job from the memory;

identify a first worker cloud computing device from a plurality of worker cloud computing devices with available resources sufficient to meet required resources for a highest-priority task associated with the computing job comprising a plurality of prioritized tasks;

determine, based upon ownership information associated with the computing job and ownership information associated with at least one other task assigned to the first worker cloud computing device, whether an ownership conflict would result from an assignment of the highest-priority task to the first worker cloud computing device; and

assign the highest-priority task to the first worker cloud computing device in response to determining that the ownership conflict would not result from the assignment of the highest-priority task to the first worker cloud computing device.

13. The system of claim 12, where the processor is further programmed to:

estimate computing resources required to complete each of the plurality of prioritized tasks; and

identify the highest-priority task to assign to the first worker cloud computing device based upon priority information associated with each of the prioritized tasks and the computing resources required to complete each of the plurality of prioritized tasks.

14. The system of claim 12, where the processor is further programmed to:

determine, based upon the ownership information associated with the computing job and the ownership information associated with the at least one other task assigned to the first worker cloud computing device, that the ownership conflict would result from the assignment of the highest-priority task to the first worker cloud computing device;

identify a second worker cloud computing device with available resources sufficient to meet the required resources for the highest-priority task without causing the ownership conflict based upon the ownership information associated with the computing job and ownership information associated with at least one other task assigned to the second worker cloud computing device; and

assign the highest-priority task to the second worker cloud computing device in response to determining that the second worker cloud computing device comprises the available resources sufficient to meet the required resources for the highest-priority task without causing the ownership conflict.

15. The system of claim 12, where, in being programmed to identify the first worker cloud computing device from the plurality of worker cloud computing devices with the available resources sufficient to meet the required resources for the highest-priority task associated with the computing job, the processor is programmed to:

query each of the plurality of worker cloud computing devices for available resources of that worker cloud computing device; and

receive a response from each of the plurality of worker cloud computing devices with the available resources of that worker cloud computing device.

16. The system of claim 15, where the available resources of each of the plurality of worker cloud computing devices comprise one of available processor resources and available input/output (I/O) resources, and where the processor is further programmed to:

determine which of processor resource requirements and I/O resource requirements for the highest-priority task is greater; and

identify the first worker cloud computing device among the plurality of worker cloud computing devices with a greatest of the one of the available processor resources and the available input/output (I/O) resources that match the determined greater of the processor resource requirements and I/O resource requirements for the highest-priority task.

17. The system of claim 12, where the processor is further programmed to:

receive updated available resources from the first worker cloud computing device in response to assigning the highest-priority task to the first worker cloud computing device;

determine which of the first worker cloud computing device and other of the plurality of worker cloud computing devices has a greatest resource availability to meet required resources for a next highest-priority task associated with the computing job;

determine, based upon the ownership information associated with the computing job and ownership information associated with at least one other task assigned to the worker cloud computing device determined to have the greatest available resources, whether an additional ownership conflict would result from an assignment of the next highest-priority task to that worker cloud computing device; and

assign the next highest-priority task to the worker cloud computing device determined to have the greatest resource availability to meet the required resources for the next highest-priority task in response to determining that the additional ownership conflict would not result from the assignment of the next highest-priority task to that worker cloud computing device.

18. The system of claim 12, where the processor is further programmed to:

iteratively assign remaining tasks associated with the computing job to additional ones of the plurality of worker cloud computing devices in decreasing order of priority and based upon highest available resources among the additional ones of the plurality of worker cloud computing devices for each iteration of task assignment without causing additional ownership conflicts.

19. A computer program product comprises a computer readable storage medium including computer readable program code, wherein the computer readable program code when executed on a computer causes the computer to:

identify a first worker cloud computing device from a plurality of worker cloud computing devices with available resources sufficient to meet required resources for a highest-priority task associated with a computing job comprising a plurality of prioritized tasks;

determine, based upon ownership information associated with the computing job and ownership information associated with at least one other task assigned to the first worker cloud computing device, whether an ownership conflict would result from an assignment of the highest-priority task to the first worker cloud computing device; and

assign the highest-priority task to the first worker cloud computing device in response to determining that the ownership conflict would not result from the assignment of the highest-priority task to the first worker cloud computing device.

20. The computer program product of claim 19, where the computer readable program code when executed on the computer further causes the computer to:

estimate computing resources required to complete each of the plurality of prioritized tasks; and

identify the highest-priority task to assign to the first worker cloud computing device based upon priority information associated with each of the prioritized tasks and the computing resources required to complete each of the plurality of prioritized tasks.

21. The computer program product of claim 19, where the computer readable program code when executed on the computer further causes the computer to:

determine, based upon the ownership information associated with the computing job and the ownership information associated with the at least one other task assigned to the first worker cloud computing device, that the ownership conflict would result from the assignment of the highest-priority task to the first worker cloud computing device;

identify a second worker cloud computing device with available resources sufficient to meet the required resources for the highest-priority task without causing the ownership conflict based upon the ownership information associated with the computing job and ownership information associated with at least one other task assigned to the second worker cloud computing device; and

assign the highest-priority task to the second worker cloud computing device in response to determining that the second worker cloud computing device comprises the available resources sufficient to meet the required resources for the highest-priority task without causing the ownership conflict.

22. The computer program product of claim 19, where, in causing the computer to identify the first worker cloud computing device from the plurality of worker cloud computing devices with the available resources sufficient to meet the required resources for the highest-priority task associated with the computing job, the computer readable program when executed on the computer causes the computer to:

query each of the plurality of worker cloud computing devices for available resources of that worker cloud computing device; and

receive a response from each of the plurality of worker cloud computing devices with the available resources of that worker cloud computing device.

23. The computer program product of claim 22, where the available resources of each of the plurality of worker cloud computing devices comprise one of available processor resources and available input/output (I/O) resources, and where the computer readable program code when executed on the computer further causes the computer to:

determine which of processor resource requirements and I/O resource requirements for the highest-priority task is greater; and

identify the first worker cloud computing device among the plurality of worker cloud computing devices with a greatest of the one of the available processor resources and the available input/output (I/O) resources that match the determined greater of the processor resource requirements and I/O resource requirements for the highest-priority task.

24. The computer program product of claim 19, where the computer readable program code when executed on the computer further causes the computer to:

receive updated available resources from the first worker cloud computing device in response to assigning the highest-priority task to the first worker cloud computing device;

determine which of the first worker cloud computing device and other of the plurality of worker cloud computing devices has a greatest resource availability to meet required resources for a next highest-priority task associated with the computing job;

determine, based upon the ownership information associated with the computing job and ownership information associated with at least one other task assigned to the worker cloud computing device determined to have the greatest available resources, whether an additional ownership conflict would result from an assignment of the next highest-priority task to that worker cloud computing device; and

assign the next highest-priority task to the worker cloud computing device determined to have the greatest resource availability to meet the required resources for the next highest-priority task in response to determining that the additional ownership conflict would not result from the assignment of the next highest-priority task to that worker cloud computing device.

25. The computer program product of claim 19, where the computer readable program code when executed on the computer further causes the computer to:

iteratively assign remaining tasks associated with the computing job to additional ones of the plurality of worker cloud computing devices in decreasing order of priority and based upon highest available resources among the additional ones of the plurality of worker cloud computing devices for each iteration of task assignment without causing additional the ownership conflicts.