Methods and Apparatus for Job State Tracking in Cluster Computing

• US 20130232164A1
• Filed: 08/30/2010
• Published: 09/05/2013
• Est. Priority Date: 08/30/2010
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Bigney, Andrey J., Reiser, Stephen Paul
• Granted Patent: US 8,996,469 B2
• Time in Patent Office: Days
• US Class Current: 707/769
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources

Abstract

Embodiments of a state tracking technique may enable real-time tracking of jobs in a computer cluster. A state object is provided that allows a job to be implemented as a distributable database. The job may be tracked while the job is processing via the state tracking technique. Using the state tracking technique, the cluster may track the location of the state objects for jobs in a database. However, only location information for the state object, and not the job metadata itself, is stored in the central database. This reduces the amount of data stored in the central database, distributing the metadata across the cluster, thus improving database performance and reducing bandwidth requirements on the network. Information about a job may be acquired via a query to the central database to find the location of the respective state object, and then a query to the state object (or to a proxy).

Claims

1. A method, comprising:

receiving, by a first compute node of a plurality of compute nodes on a cluster computing system, a state object for a job to be executed by a distributed application on the cluster computing system, wherein the state object includes a job metadata database that stores status and history information for the respective job as executed by the distributed application on the plurality of compute nodes of the cluster computing system, wherein the cluster computing system comprises at least the plurality of compute nodes connected together through a network;

sending, by the first compute node through the network, a location indicator for the state object on the first compute node to a state tracker at a second compute node of the plurality of compute nodes on the cluster computing system, the state tracker recording that the state object is located at the first compute node in a state object location database for the cluster computing system; and

updating the job metadata database of the state object according to execution of one or more tasks of the job on the first compute node.

2. The method as recited in claim 1, further comprising:

transferring the state object from the first compute node to a third compute node of the plurality of compute nodes on the cluster computing system;

sending a location indicator for the state object on the third compute node to the state tracker in response to said transfer; and

the state tracker updating the state object location database according to the location indicator for the state object on the third compute node.

3. The method as recited in claim 2, wherein said sending a location indicator for the state object on the third compute node to the state tracker is performed by the sending compute node.

4. The method as recited in claim 1, further comprising:

receiving, by the state tracker, a query requesting a location on the network of the state object;

obtaining, by the state tracker, the location indicator for the state object from the state object location database; and

sending, by the state tracker, a response to the query, wherein the response includes the location indicator for the state object obtained from the state object location database.

5. The method as recited in claim 4, further comprising an initiator of the query remotely querying the state object via the location indicator for the state object indicated by the response to request job metadata for the job.

6. The method as recited in claim 1, further comprising:

receiving, by the state tracker, a query requesting a location on the network of the state object;

obtaining, by the state tracker, a proxy to the state object;

sending, by the state tracker, a response to the query indicating the proxy to the state object; and

an initiator of the query remotely querying the state object for the job via the proxy to the state object indicated by the response to request job metadata.

21. The method as recited in claim 1, wherein the state object comprises a single file.

7. A cluster computing system, comprising:

a plurality of compute nodes coupled to a network;

a node coupled to the network that implements a state tracker; and

a storage that stores a state object location database for the cluster computing system;

wherein, during operation, a first compute node of the a plurality of compute nodes;

receives a state object for a job to be executed by a distributed application on the cluster computing system, wherein the state object includes a job metadata database that stores status and history information for the respective job as executed by the distributed application on the cluster computing system;

sends a location indicator for the state object on the first compute node to the state tracker; and

updates the job metadata database of the state object according to execution of one or more tasks of the job on the first compute node;

wherein, during operation, the state tracker;

receives the location indicator for the state object from the first compute node; and

records the location indicator for the state object in the state object location database.

8. The cluster computing system as recited in claim 7, wherein said sending a location indicator for the state object on the first compute node to a state tracker is performed by the state object on the first compute node.

9. The cluster computing system as recited in claim 7,

wherein the first compute node transfers the state object to a second node of the plurality of compute nodes on the cluster computing system, and wherein a location indicator for the state object on the second compute node is sent to the state tracker in response to said transfer;

and

wherein the state tracker updates the state object location database with according to the location indicator for the state object on the second compute node.

10. The cluster computing system as recited in claim 9, wherein said sending a location indicator for the state object on the second compute node to the state tracker is performed by the first compute node.

11. The cluster computing system as recited in claim 7, wherein, during operation, the state tracker:

receives a query requesting a location on the network of the state object;

obtains the location indicator for the state object from the state object location database; and

sends a response to the query, wherein the response includes the location indicator for the state object obtained from the state object location database.

12. The cluster computing system as recited in claim 7, wherein, during operation, the state tracker:

receives a query requesting a location on the network of the state object;

obtains a proxy to the state object; and

sends a response to the query indicating the proxy to the state object.

13. A non-transitory computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to implement:

receiving, by a first compute node of a plurality of compute nodes on a cluster computing system, a state object for a job to be executed by a distributed application on the cluster computing system, wherein the state object includes a job metadata database that stores status and history information for the respective job as executed by the distributed application on the cluster computing system;

sending, by the state object, a location indicator for the state object on the first compute node to a state tracker on a second compute node of the cluster computing system;

the state tracker recording the location indicator for the state object in a state object location database for the cluster computing system; and

updating the job metadata database of the state object according to execution of one or more tasks of the job on the first compute node.

14. The non-transitory computer-readable storage medium as recited in claim 13, wherein the program instructions are computer-executable to implement:

transferring the state object from the first compute node to a third compute node of the plurality of compute nodes on the cluster computing system;

sending a location indicator for the state object on the third compute node to the state tracker in response to said transfer; and

the state tracker updating the state object location database according to the location indicator for the state object on the third compute node.

15. The non-transitory computer-readable storage medium as recited in claim 14, wherein said sending a location indicator for the state object on the third compute node to the state tracker is performed by the first compute node.

16. The non-transitory computer-readable storage medium as recited in claim 13, wherein the program instructions are computer-executable to implement:

receiving, by the state tracker, a query requesting a location on the network of the state object;

obtaining, by the state tracker, the location indicator for the state object from the state object location database; and

sending, by the state tracker, a response to the query, wherein the response includes the location indicator for the state object obtained from the state object location database.

17. The non-transitory computer-readable storage medium as recited in claim 16, wherein the program instructions are computer-executable to implement an initiator of the query remotely querying the state object via the location indicator for the state object indicated by the response to request job metadata for the job.

18. The non-transitory computer-readable storage medium as recited in claim 13, wherein the program instructions are computer-executable to implement:

receiving, by the state tracker, a query requesting a location on the network of the state object;

obtaining, by the state tracker, a proxy to the state object; and

sending, by the state tracker, a response to the query indicating the proxy to the state object; and

an initiator of the query remotely querying the state object for the job via the proxy to the state object indicated by the response to request job metadata.

19. A node of a cluster computing system, comprising:

at least one processor; and

a memory comprising program instructions, wherein the program instructions are executable by the at least one processor to;

receive, from a first compute node of a plurality of compute nodes on the cluster computing system, a location indicator for a state object for a job to be executed by a distributed application on the cluster computing system, wherein the state object includes a job metadata database that stores status and history information for the respective job˜

executed by the distributed application on the cluster computing system, and wherein the state object is configured to be transferred between compute nodes on the cluster computing system during execution of the job by the distributed application;

record the location indicator for the state object as the current location of the state object in a state object location database for the cluster computing system; and

respond to queries requesting the current location of the state object according to the current location of the state object in the state object location database.

20. The node as recited in claim 19, wherein the program instructions are executable by the at least one processor to:

receive a message indicating that the state object has been transferred from the first compute node to a second compute node of the plurality of compute nodes on the cluster computing system, wherein the message includes a location indicator for the state object on the second compute node; and

updating the current location of the state object in the state object location database according to the location indicator for the state object on the second compute node.