Distributed Job Manager Recovery

• US 20080307258A1
• Filed: 06/11/2007
• Published: 12/11/2008
• Est. Priority Date: 06/11/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Giles, James R., Challenger, James R., Jacques De Silva, Gabriela, Challenger, Louis R.
• Granted Patent: US 7,779,298 B2
• Time in Patent Office: Days
• US Class Current: 714/20
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality

Abstract

A method is provided for the recovery of an instance of a job manager running on one of a plurality of nodes used to execute the processing elements associated with jobs that are executed within a cooperative data stream processing system. The states of the processing elements are checkpointed to a persistence mechanism in communication with the job manager. From the checkpointed processing element states, the state of each distributed job is determined and checkpointed. Processing element states are also checkpointed locally to the nodes one which the processing elements are running. Following a failure of the job manager, the job manager is reinstantiated on one of the nodes. The recovery instance of the job manger obtains state data for processing elements and jobs from the persistence mechanism and constructs an initial state for jobs and processing elements. These initial states are reconciled against the current states of the processing elements and adjustments are made accordingly. Once the job and processing element states are reconciled, the system is returned to normal operation.

Claims

1. A method for failure recovery of a job manager within a data processing system, the method comprising:

instantiating a job manager on one of a plurality of nodes to manage distributed execution of jobs on the plurality of nodes, wherein management of job execution for each job comprises managing the deployment and execution of a plurality of processing elements associated with each job on the plurality of nodes in accordance with instructions from a job scheduler;

checkpointing a current state associated with each one of the plurality of processing elements associated with the jobs managed by the job manager; and

recovering the job manager subsequent to a failure of the job manager using the checkpointed processing element states.

2. The method of claim 1, wherein the step of checkpointing the current state further comprises checkpointing the current state of each processing element in response to a state change.

3. The method of claim 1, wherein the step of checkpointing the current state further comprises saving the current state of each one of the plurality of processing elements to a persistence mechanism in communication with the job manager.

4. The method of claim 3, wherein the persistence mechanism comprises at least one database in communication with the job manager.

5. The method of claim 1, wherein the step of recovering the job manager instance further comprises:

instantiating a replacement job manager;

retrieving the checkpointed processing element states using the replacement job manager; and

constructing an initial recovered state for each job and its constituent processing elements using the retrieved checkpointed processing element states.

6. The method of claim 5, wherein the step of recovering the job manager instance further comprises reconciling the initial recovered states for the plurality of processing elements with actual current states for the plurality of processing elements.

7. The method of claim 1, further comprising:

suspending checkpointing of the current state for each one of the plurality of processing elements in response to the failure of the job manager; and

resuming checkpointing of the current state for each one of the plurality of processing elements following recovery of the job manager.

8. The method of claim 7, wherein the step of suspending checkpointing further comprises maintaining a chronological log of checkpointed states at each one of the plurality of processing elements;

and

the step of resuming checkpointing further comprises delivering the chronological log of checkpointed states from each one of the plurality of processing elements to the recovered job manager.

9. The method of claim 1, further comprising checkpointing the current state associated with each one of the plurality of processing elements at each node on which the plurality of processing elements are being executed.

10. The method of claim 1, further comprising determining a current job state for each job managed by the job manager using the checkpoint processing element states;

and

saving the current job states in a persistence mechanism in communication with the job manager.

11. The method of claim 10, further comprising updating each saved current job state in response to changes in the checkpointed processing element states.

12. The method of claim 10, wherein the step of recovering the job manager further comprises using the saved current job states.

13. The method of claim 12, wherein the step of using the saved current job states further comprises identifying uncompleted jobs based on the saved current job states.

14. The method of claim 1, wherein the current job states comprise submitted, mapping, dispatching, canceling and running.

15. The method of claim 1, further comprising accounting for failures in at least one of processing elements, execution nodes, a database management system and a dataflow graph manager that are concurrent with the job manager failure.

16. The method of claim 1, wherein the checkpointed current states comprise starting a job program element, stopping a job program element, suspending a job program element, resuming a job program element, loading a job program element, unloading a job program element, node failure, node controller failure, master node controller failure, node boot, node controller boot, master node controller boot, job manager failure or job manager boot.

17. A method for failure recovery of a job manager within a data processing system, the method comprising:

instantiating a job manager on one of a plurality of nodes to manage distributed execution of jobs on the plurality of nodes, wherein management of job execution for each job comprises managing the deployment and execution of a plurality of processing elements associated with each job on the plurality of nodes in accordance with instructions from a job scheduler;

checkpointing a current state associated with each job;

checkpointing a current state associated with each one of the plurality of processing elements associated with each job managed by the job manager;

determining a current job state for each job managed by the job manager using the checkpointed job and processing element states; and

recovering the job manager subsequent to a failure of the job manager using the checkpointed processing element states, the checkpointed job states and the current state for each job.

18. A method for failure recovery of a job manager within a distributed processing system, the method comprising:

instantiating a job manager on one of a plurality of nodes to manage distributed execution of jobs on the plurality of nodes, wherein management of job execution for each job comprises managing the deployment and execution of a plurality of processing elements associated with each job on the plurality of nodes in accordance with instructions from a job scheduler;

checkpointing a current state associated with each job;

checkpointing a current state associated with each one of the plurality of processing elements associated with the jobs managed by the job manager; and

recovering the job manager subsequent to a failure of the job manager using the checkpointed job and processing element states.

19. The method of claim 18, wherein the step of recovery the job manager further comprises coordinating recovery of the job manager with at least one component responsible for managing a given type of processing element.

20. The method of claim 19, wherein the component comprises a dataflow graph manager, a system service manager or a virtual world manager.