METHODS AND APPARATUS FOR RESOURCE MANAGEMENT IN CLUSTER COMPUTING

• US 20140143787A1
• Filed: 01/27/2014
• Published: 05/22/2014
• Est. Priority Date: 08/30/2010
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Bostic, Sandford P., Reiser, Stephen Paul, Bigney, Andrey J.
• Granted Patent: US 9,262,218 B2
• Time in Patent Office: Days
• US Class Current: 718/104
• CPC Class Codes: G06F 11/30 Monitoring
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 2209/5017 Task decomposition
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: G06F 9/5072 Grid computing

Abstract

Embodiments of an event-driven resource management technique may enable the management of cluster resources at a sub-computer level (e.g., at the thread level) and the decomposition of jobs at an atomic (task) level. A job queue may request a resource for a job from a resource manager, which may locate a resource in a resource list and grant the resource to the job queue. After the resource is granted, the job queue sends the job to the resource, on which the job may be partitioned into tasks and from which additional resources may be requested from the resource manager. The resource manager may locate additional resources in the list and grant the resources to the resource. The resource sends the tasks to the granted resources for execution. As resources complete their tasks, the resource manager is informed so that the status of the resources in the list can be updated.

Claims

1-20. -20. (canceled)

21. A method for tracking jobs performed by computing nodes of a cluster computing system, the method comprising:

monitoring, by a management system, a plurality of computing nodes and an availability of resources in the cluster computing system;

identifying, by the management system, that at least one computing node of the plurality of computing nodes is available for performing a job submitted to a job queue;

generating a job state object for tracking a job status of the job submitted to the job queue; and

updating the job state object by a system independent of the management system upon completion of at least one task of the job state object.

22. The method of claim 21, further comprising:

generating a task state object for tracking a task status for the at least one task;

providing the task state object to the at least one computing node; and

updating the task state object in response to completing the task.

23. The method of claim 21, wherein the system independent of the management system comprises the at least one computing node and further comprising transmitting, by a job scheduling system, the job state object to the at least one computing node.

24. The method of claim 23, further comprising

transmitting, by the at least one computing node, job metadata extracted from the job state object in response to a job status query from the job scheduling system, wherein the job metadata is indicative of the updated job status.

25. The method of claim 24, wherein the job state object comprises a database storing the job metadata indicative of the job status as a plurality of records in the database and application code executable by the at least one computing node to receive database queries and retrieve a subset of the plurality of the records from the database in response to the job status query.

26. The method of claim 23, wherein the system independent of the management system further comprises at least one additional computing node of the plurality of computing nodes and further comprising:

transmitting the updated job state object to the at least one additional computing node for performing at least one additional task associated with the job;

updating, by the at least one additional computing node, the job state object subsequent to performing the at least one additional task associated with the job.

27. The method of claim 21, further comprising:

partitioning the job into a plurality of tasks;

partitioning the job state object into a plurality of task state objects, wherein each of the plurality of task state objects is configured for tracking a respective one of the plurality of tasks;

distributing the plurality of tasks and the plurality of task state objects to at least some computing nodes from the plurality of computing nodes; and

updating, by each of the at least some computing nodes, a respective task state object subsequent to performing a respective task.

28. A system for tracking jobs performed by computing nodes of a cluster computing system, the system comprising:

a management system configured for;

monitoring a plurality of computing nodes and an availability of resources in the cluster computing system, and

identifying that at least one computing node of the plurality of computing nodes is available for performing a job submitted to a job queue; and

at least one computing system in communication with and independent of the management system, the at least one system configured for;

generating a job state object for tracking a job status of the job submitted to the job queue, and

updating the job state object by a system the management system upon completion of at least one task of the job state object.

29. The system of claim 28, wherein the at least one computing system is further configured for:

generating a task state object for tracking a task status for the at least one task;

providing the task state object to the at least one computing node; and

updating the task state object in response to completing the task.

30. The system of claim 28, wherein the at least one computing system comprises the at least one computing node and a job scheduling system configured for transmitting the job state object to the at least one computing node.

31. The system of claim 30, wherein the at least one computing node is configured for transmitting job metadata extracted from the job state object in response to a job status query from the job scheduling system, wherein the job metadata is indicative of the updated job status.

32. The system of claim 31, wherein the job state object comprises a database storing the job metadata indicative of the job status as a plurality of records in the database and application code executable by the at least one computing node to receive database queries and retrieve a subset of the plurality of the records from the database in response to the job status query.

33. The system of claim 30, wherein the at least one computing system further comprises at least one additional computing node of the plurality of computing nodes, wherein the at least one additional computing node is configured for:

transmitting the updated job state object to the at least one additional computing node for performing at least one additional task associated with the job;

updating the job state object subsequent to performing the at least one additional task associated with the job.

34. The system of claim 28, wherein the at least one system is further configured for:

partitioning the job into a plurality of tasks;

partitioning the job state object into a plurality of task state objects, wherein each of the plurality of task state objects is configured for tracking a respective one of the plurality of tasks; and

distributing the plurality of tasks and the plurality of task state objects to at least some computing nodes from the plurality of computing nodes, wherein each of the at least some computing nodes is configured for updating a respective task state object subsequent to performing a respective task.

35. A non-transitory computer-readable medium having program code stored thereon that is executable by a processor for tracking jobs performed by computing nodes of a cluster computing system, the program code comprising:

program code for monitoring, by a management system, a plurality of computing nodes and an availability of resources in the cluster computing system;

program code for identifying, by the management system, that at least one computing node of the plurality of computing nodes is available for performing a job submitted to a job queue;

program code for generating a job state object for tracking a job status of the job submitted to the job queue; and

program code for updating the job state object by a system independent of the management system upon completion of at least one task of the job state object.

36. The non-transitory computer-readable medium of claim 35, wherein the program code further comprises:

program code for generating a task state object for tracking a task status for the at least one task;

program code for providing the task state object to the at least one computing node; and

program code for updating the task state object in response to completing the task.

37. The non-transitory computer-readable medium of claim 35, wherein the system independent of the management system comprises the at least one computing node and wherein the program code further comprises program code for transmitting, by a job scheduling system, the job state object to the at least one computing node.

38. The non-transitory computer-readable medium of claim 37, wherein the program code further comprises program code for transmitting, by the at least one computing node, job metadata extracted from the job state object in response to a job status query from the job scheduling system, wherein the job metadata is indicative of the updated job status.

39. The non-transitory computer-readable medium of claim 37, wherein the job state object comprises:

a database storing the job metadata indicative of the job status as a plurality of records in the database; and

application code executable by the at least one computing node to receive database queries and retrieve a subset of the plurality of the records from the database in response to the job status query.

40. The non-transitory computer-readable medium of claim 37, wherein the system independent of the management system further comprises at least one additional computing node of the plurality of computing nodes and wherein the program code further comprises:

program code for transmitting the updated job state object to the at least one additional computing node for performing at least one additional task associated with the job;

program code for updating, by the at least one additional computing node, the job state object subsequent to performing the at least one additional task associated with the job.