Methods and apparatus for resource management in cluster computing

• US 8,640,137 B1
• Filed: 08/30/2010
• Issued: 01/28/2014
• Est. Priority Date: 08/30/2010
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Reiser, Stephen Paul, Bigney, Andrey J., Bostic, Sandford P.
• Primary Examiner(s): Puente, Emerson
• Assistant Examiner(s): PATEL, HIREN P
• Application Number:
  US12/871,699
• Time in Patent Office: 1,247 Days
• Field of Search: None
• US Class Current: 718/104
• CPC Class Codes: G06F 11/30 Monitoring
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 2209/5017 Task decomposition
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: G06F 9/5072 Grid computing

Abstract

Embodiments of an event-driven resource management technique may enable the management of cluster resources at a sub-computer level (e.g., at the thread level) and the decomposition of jobs at an atomic (task) level. A job queue may request a resource for a job from a resource manager, which may locate a resource in a resource list and grant the resource to the job queue. After the resource is granted, the job queue sends the job to the resource, on which the job may be partitioned into tasks and from which additional resources may be requested from the resource manager. The resource manager may locate additional resources in the list and grant the resources to the resource. The resource sends the tasks to the granted resources for execution. As resources complete their tasks, the resource manager is informed so that the status of the resources in the list can be updated.

Claims

1. A method for tracking jobs performed by computing nodes of a cluster computing system, the method comprising:

transmitting, by a job scheduling system, a job description via a data network to a resource tracking system for the cluster computing system, wherein the job description specifies a number of resources for performing a job comprising a plurality of tasks, wherein the resource tracking system maintains a resource list comprising, for each resource of a plurality of resources, a respective availability of the resource and a respective network location identifying a respective computing node in the cluster computing system at which the resource is located;

determining, by the resource tracking system, that a subset of resources from the plurality of resources is available in response to receiving the job description, wherein an availability of the subset of resources is determined by reference to the resource list;

transmitting, by the resource tracking system, at least one network identifier via the data network to the job scheduling system, wherein the at least one network identifier identifies at least one computing node at which the subset of resources are located;

generating, by the job scheduling system, a job state object for tracking a job status for the job;

transmitting, by the job scheduling system, the job state object and the job to the at least one computing node;

updating, by the at least one computing node, the job state object to describe an updated job status subsequent to performing at least one task of the plurality of tasks;

transmitting, by the at least one computing node, job metadata extracted from the job state object in response to a job status query from the job scheduling system, wherein the job metadata indicates the updated job status; and

transmitting, by the at least one computing node, the updated job state object to at least one additional computing node for performing at least one additional task associated with the job;

wherein the at least one additional computing node updates the job state object subsequent to performing the at least one additional task and notifies the job scheduling system of the update to the job state object by the at least one additional computing node.

2. The method of claim 1, wherein the job state object comprises a database storing the job metadata as a plurality of records in the database, wherein the job status query comprises a database query, and wherein transmitting the job metadata comprises:

retrieving a subset of the plurality of records from the database, wherein the subset comprises information specified by the database query; and

transmitting the subset to the job scheduling system.

3. The method of claim 2, wherein the job state object further comprises application code executable by the at least one computing node to receive database queries and retrieve the subset from the database in response to the database query.

4. The method of claim 1, further comprising:

transmitting, by the job scheduling system, the network identifier for the at least one computing node to a state tracking system of the cluster computing system;

transmitting, by the state tracking system, the network identifier to a requestor system in response to a request for a network location of the job state object from the requestor system external to the cluster computing system; and

transmitting, by the at least one computing node, the job metadata extracted from the job state object and describing the updated job status to the job scheduling system in response to an additional job status query from the requestor system.

5. The method of claim 1, further comprising:

transmitting, by the at least one computing node, a request for additional resources for the job to the resource tracking system via the data network;

determining, by the resource tracking system, that at least one additional resource is available from the resource list in response to receiving the request for additional resources;

transmitting, by the resource tracking system, at least one additional network identifier to the at least one computing node via the data network, wherein the at least one additional network identifier identifies the at least one additional computing node at which the additional resource is located; and

responsive to receiving the at least one additional network identifier, transmitting the job state object and the at least one additional task of the plurality of tasks from the at least one computing node to the at least one additional computing node.

6. The method of claim 1, further comprising:

partitioning, by the at least one computing node, the job into a plurality of tasks;

partitioning, by the at least one computing node, the job state object into a plurality of task state objects, wherein each of the plurality of task state objects is configured for tracking a respective one of the plurality of tasks;

distributing, from the at least one computing node, the plurality of tasks and the plurality of task state objects to a plurality of additional computing nodes; and

updating, by each of the plurality of additional computing nodes, a respective task state object subsequent to performing a respective task.

7. A non-transitory computer-readable medium having program code stored thereon that is executable by a processor to track jobs performed by computing nodes of a cluster computing system, the non-transitory computer-readable medium comprising:

program code for transmitting, by a job scheduling system, a job description via a data network to a resource tracking system for the cluster computing system, wherein the job description specifies a number of resources for performing a job comprising a plurality of tasks, wherein the resource tracking system maintains a resource list comprising, for each resource of a plurality of resources, a respective availability of the resource and a respective network location identifying a respective computing node in the cluster computing system at which the resource is located;

program code for determining, by the resource tracking system, that a subset of resources from the plurality of resources is available in response to receiving the job description, wherein an availability of the subset of resources is determined by reference to the resource list;

program code for transmitting, by the resource tracking system, at least one network identifier via the data network to the job scheduling system, wherein the at least one network identifier identifies at least one computing node at which the subset of resources are located;

program code for generating, by the job scheduling system, a job state object for tracking a job status for the job;

program code for transmitting, by the job scheduling system, the job state object and the job to the at least one computing node;

program code for updating, by the at least one computing node, the job state object to describe an updated job status subsequent to performing at least one task of the plurality of tasks;

program code for transmitting, by the at least one computing node, job metadata extracted from the job state object in response to a job status query from the job scheduling system, wherein the job metadata indicates the updated job status;

program code for transmitting, by the at least one computing node, the updated job state object to at least one additional computing node for performing at least one additional task associated with the job;

program code for updating the job state object by the at least one additional node computing node subsequent to performing the at least one additional task; and

program code for notifying, by the at least one additional computing node, the job scheduling system of the update to the job state object by the at least one additional computing node.

8. The non-transitory computer-readable medium of claim 7, wherein the job state object comprises a database storing the job metadata as a plurality of records in the database, wherein the job status query comprises a database query, and wherein transmitting the job metadata comprises:

retrieving a subset of the plurality of records from the database, wherein the subset comprises information specified by the database query; and

transmitting the subset to the job scheduling system.

9. The non-transitory computer-readable medium of claim 8, wherein the job state object further comprises application code executable by the at least one computing node to receive database queries and retrieve the subset from the database in response to the database query.

10. The non-transitory computer-readable medium of claim 7, further comprising:

transmitting, by the job scheduling system, the network identifier for the at least one computing node to a state tracking system of the cluster computing system;

transmitting, by the state tracking system, the network identifier to a requestor system in response to a request for a network location of the job state object from the requestor system external to the cluster computing system; and

transmitting, by the at least one computing node, the job metadata extracted from the job state object and describing the updated job status to the job scheduling system in response to an additional job status query from the requestor system.

11. The non-transitory computer-readable medium of claim 7, further comprising:

program code for transmitting, by the at least one computing node, a request for additional resources for the job to the resource tracking system via the data network;

program code for determining, by the resource tracking system, that at least one additional resource is available from the resource list in response to receiving the request for additional resources;

program code for transmitting, by the resource tracking system, at least one additional network identifier to the at least one computing node via the data network, wherein the at least one additional network identifier identifies the at least one additional computing node at which the additional resource is located; and

program code for responding to receiving the at least one additional network identifier by transmitting the job state object and the at least one additional task of the plurality of tasks from the at least one computing node to the at least one additional computing node.

12. The non-transitory computer-readable medium of claim 7, further comprising:

program code for partitioning, by the at least one computing node, the job into a plurality of tasks;

program code for partitioning, by the at least one computing node, the job state object into a plurality of task state objects, wherein each of the plurality of task state objects is configured for tracking a respective one of the plurality of tasks;

program code for distributing, from the at least one computing node, the plurality of tasks and the plurality of task state objects to a plurality of additional computing nodes; and

program code for updating, by each of the plurality of additional computing nodes, a respective task state object subsequent to performing a respective task.

13. A cluster computing system comprising:

a job scheduling system comprising a first processor, the first processor configured for;

transmitting a job description via a data network to a resource tracking system for the cluster computing system, wherein the job description specifies a number of resources for performing a job comprising a plurality of tasks, wherein the resource tracking system maintains a resource list comprising, for each resource of a plurality of resources, a respective availability of the resource and a respective network location identifying a respective computing node in the cluster computing system at which the resource is located;

generating a job state object for tracking a job status for the job, and

transmitting the job state object and the job to at least one computing node;

a resource tracking system in communication with the job scheduling system via the data network, wherein the resource tracking system comprises a second processor configured for;

determining that a subset of resources from the plurality of resources is available in response to receiving the job description, wherein an availability of the subset of resources is determined by reference to the resource list;

transmitting at least one network identifier via the data network to the job scheduling system, wherein the at least one network identifier identifies the at least one computing node at which the subset of resources are located;

the at least one computing node in communication with the job scheduling system via the data network, wherein the at least one computing node comprises a third processor configured for;

updating the job state object to describe an updated job status subsequent to performing at least one task of the plurality of tasks,

transmitting job metadata extracted from the job state object in response to a job status query from the job scheduling system, wherein the job metadata indicates the updated job status, and

transmitting, by the at least one computing node, the updated job state object to at least one additional computing node for performing at least one additional task associated with the job; and

the at least one additional computing node in communication with the at least one computing node via the data network, wherein the at least one additional computing node comprises a fourth processor configured for;

updating the job state object subsequent to performing the at least one additional task, and

notifying the job scheduling system of the update to the job state object by the at least one additional computing node.

14. The cluster computing system of claim 13, wherein the job state object comprises a database storing the job metadata as a plurality of records in the database, wherein the job status query comprises a database query, and wherein transmitting the job metadata comprises:

retrieving a subset of the plurality of records from the database, wherein the subset comprises information specified by the database query; and

transmitting the subset to the job scheduling system.

15. The cluster computing system of claim 14, wherein the job state object further comprises application code executable by the at least one computing node to receive database queries and retrieve the subset from the database in response to the database query.