Methods and apparatus for resource management in cluster computing

• US 10,067,791 B2
• Filed: 01/04/2016
• Issued: 09/04/2018
• Est. Priority Date: 08/30/2010
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Bostic, Sandford P., Reiser, Stephen Paul, Bigney, Andrey J.
• Primary Examiner(s): Patel, Hiren
• Application Number:
  US14/987,406
• Publication Number:
  US 20160117194A1
• Time in Patent Office: 974 Days
• Field of Search: None
• CPC Class Codes: G06F 11/30 Monitoring
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 2209/5017 Task decomposition
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: G06F 9/5072 Grid computing

Abstract

Embodiments of an event-driven resource management technique may enable the management of cluster resources at a sub-computer level (e.g., at the thread level) and the decomposition of jobs at an atomic (task) level. A job queue may request a resource for a job from a resource manager, which may locate a resource in a resource list and grant the resource to the job queue. After the resource is granted, the job queue sends the job to the resource, on which the job may be partitioned into tasks and from which additional resources may be requested from the resource manager. The resource manager may locate additional resources in the list and grant the resources to the resource. The resource sends the tasks to the granted resources for execution. As resources complete their tasks, the resource manager is informed so that the status of the resources in the list can be updated.

Claims

1. A method for tracking jobs performed by computing nodes of a cluster computing system, the method comprising:

monitoring, by a management computer, a plurality of computing nodes and an availability of resources provided by the plurality of computing nodes in the cluster computing system;

identifying, by the management computer, a first computing node of the plurality of computing nodes that is available for performing a first job submitted to a job queue;

identifying, by the management computer, a second computing node of the plurality of computing nodes that is available for performing a second job submitted to the job queue;

generating a first job state object specific to the first job for tracking a first job status of the first job and a second job state object specific to the second job for tracking a second job status of the second job;

providing the first job state object to the first computing node and the second job state object to the second computing node;

updating, after completion of a task of the first job, the first job state object independently of any updates to the second job state object after completion of a task of the second job; and

updating, after completion of the task of the second job, the second job state object independently of any updates to the first job state object after completion of the task of the first job.

2. The method of claim 1, further comprising:

generating a task state object for tracking a task status for the task of the first job;

providing the task state object to the first computing node; and

updating the task state object in response to completing the task.

3. The method of claim 1, wherein a first computer independent of the management computer updates the first job state object and a second computer independent of the management computer updates the second job state object, wherein the method further comprises transmitting, by a job scheduling system, the first job state object to the first computer and the second job state object to the second computer.

4. The method of claim 3, further comprising transmitting, by at least one of the first computer and the second computer, job metadata extracted from at least one of the first job state object and the second job state object in response to a job status query from the job scheduling system, wherein the job metadata is indicative of an updated job status for at least one of the first job and the second job.

5. The method of claim 4, wherein the first job state object comprises a first database storing first job metadata indicative of the first job status as a plurality of records in the first database and application code executable by the first computer to receive database queries and retrieve a subset of the plurality of records from the first database in response to the job status query,

wherein the second job state object comprises a second database storing second job metadata indicative of the second job status as a plurality of records in the second database and application code executable by the second computer to receive database queries and retrieve a subset of the plurality of records from the second database in response to the job status query.

6. The method of claim 3, further comprising:

transmitting the updated first job state object from the first computer to an additional computer independent of the management computer for performing an additional task associated with the first job;

updating, by the additional computer, the first job state object subsequent to performing the additional task associated with the first job.

7. The method of claim 1, further comprising:

partitioning each of the first job and the second job into a plurality of tasks;

partitioning each of the first job state object and the second job state object into a plurality of task state objects, wherein each of the plurality of task state objects is configured for tracking a respective one of the plurality of tasks;

distributing the plurality of tasks and the plurality of task state objects to at least some computing nodes from the plurality of computing nodes; and

updating, by each of the at least some computing nodes, a respective task state object subsequent to performing a respective task.

8. The method of claim 1, wherein the first job state object comprises first application code executable by the first computing node to perform operations for updating the first job state object after the completion of the task of the first job, wherein the second job state object comprises second application code executable by the second computing node to perform operations for updating the second job state object after the completion of the task of the second job.

9. The method of claim 1, wherein the first job state object is a first stand-alone object and the second job state object is a second stand-alone object.

10. A system for tracking jobs performed by computing nodes of a cluster computing system, the system comprising:

a management computer configured for;

monitoring a plurality of computing nodes and an availability of resources provided by the plurality of computing nodes in the cluster computing system,

identifying a first computing node of the plurality of computing nodes that is available for performing a first job submitted to a job queue, and

identifying a second computing node of the plurality of computing nodes that is available for performing a second job submitted to the job queue; and

at least one computer in communication with the management computer and independent of the management computer, the at least one computer configured for;

generating a first job state object specific to the first job for tracking a first job status of the first job and a second job state object specific to the second job for tracking a second job status of the second job,

providing the first job state object to the first computing node and the second job state object to the second computing node,

updating, after completion of a task of the first job, the first job state object independently of any updates to the second job state object after completion of a task of the second job, and

updating, after completion of the task of the second job, the second job state object independently of any updates to the first job state object after completion of the task of the first job.

11. The system of claim 10, wherein the at least one-computer is further configured for:

generating a task state object for tracking a task status for the task of the first job;

providing the task state object to the first computing node; and

updating the task state object in response to completing the task.

12. The system of claim 10, wherein the at least one computer comprises:

a first computer that is configured for updating the first job state object; and

a second computer that is configured for updates the second job state object,

wherein the system further comprises a job scheduling system configured for transmitting the first job state object to the first computer and the second job state object to the second computer.

13. The system of claim 12, wherein at least one of the first computer and the second computer is configured for transmitting job metadata extracted from at least one of the first job state object and the second job state object in response to a job status query from the job scheduling system, wherein the job metadata is indicative of an updated job status for at least one of the first job and the second job.

14. The system of claim 13, wherein the first job state object comprises a first database storing first job metadata indicative of the first job status as a plurality of records in the first database and application code executable by the first computer to receive database queries and retrieve a subset of the plurality of records from the first database in response to the job status query,

wherein the second job state object comprises a second database storing second job metadata indicative of the second job status as a plurality of records in the second database and application code executable by the second computer to receive database queries and retrieve a subset of the plurality of records from the second database in response to the job status query.

15. The system of claim 12, wherein the first computer is configured for transmitting the updated first job state object to an additional computer independent of the management computer for performing an additional task associated with the first job;

wherein the system further comprises the additional computer, wherein the additional computer is configured for updating the first job state object subsequent to performing the additional task associated with the first job.

16. The system of claim 10, the at least one computer is further configured for:

partitioning each of the first job and the second job into a plurality of tasks;

partitioning each of the first job state object and the second job state object into a plurality of task state objects, wherein each of the plurality of task state objects is configured for tracking a respective one of the plurality of tasks; and

distributing the plurality of tasks and the plurality of task state objects to at least some computing nodes from the plurality of computing nodes,

wherein each of the at least some computing nodes is configured for updating a respective task state object subsequent to performing a respective task.

17. A non-transitory computer-readable medium having program code stored thereon that is executable by a processor for tracking jobs performed by computing nodes of a cluster computing system, the program code comprising:

program code for monitoring, by a management computer, a plurality of computing nodes and an availability of resources provided by the plurality of computing nodes in the cluster computing system;

program code for identifying, by the management computer, a first computing node of the plurality of computing nodes that is available for performing a first job submitted to a job queue;

program code for identifying, by the management computer, a second computing node of the plurality of computing nodes that is available for performing a second job submitted to the job queue;

program code for generating a first job state object specific to the first job for tracking a first job status of the first job and a second job state object specific to the second job for tracking a second job status of the second job;

program code for providing the first job state object to the first computing node and the second job state object to the second computing node;

program code for updating, after completion of a task of the first job, the first job state object independently of any updates to the second job state object after completion of a task of the second job;

program code for updating, after completion of the task of the second job, the second job state object independently of any updates to the first job state object after completion of the task of the first job.

18. The non-transitory computer-readable medium of claim 17, wherein the program code further comprises:

program code for generating a task state object for tracking a task status for the task of the first job;

program code for providing the task state object to the first computing node; and

program code for updating the task state object in response to completing the task.

19. The non-transitory computer-readable medium of claim 17, wherein the program code further comprises program code for transmitting, by a job scheduling system, the first job state object to a first computer independent of the management computer and the second job state object to a second computer independent of the management computer.

20. The non-transitory computer-readable medium of claim 19, wherein the program code further comprises program code for transmitting, by at least one of the first computer and the second computer, job metadata extracted from at least one of the first job state object and the second job state object in response to a job status query from the job scheduling system, wherein the job metadata is indicative of an updated job status for at least one of the first job and the second job.