System and method for modifying execution of scripts for a job scheduler using deontic logic

• US 8,266,477 B2
• Filed: 01/11/2010
• Issued: 09/11/2012
• Est. Priority Date: 01/09/2009
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.), Colin Banger, Jody Steele, Tomas Gray
• Inventors: Mankovskii, Serguei, Banger, Colin, Steele, Jody, Gray, Tomas
• Primary Examiner(s): KO, CHAE M
• Application Number:
  US12/685,357
• Publication Number:
  US 20100211815A1
• Time in Patent Office: 974 Days
• Field of Search: 714/38.1, 714/25, 714/2, 714/15
• US Class Current: 714/15
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/0748 in a remote unit communicating with a single-box computer node experiencing an error/fault remote testing G06F11/2294
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality

Abstract

A system and method for modifying execution scripts associated with a job scheduler may include monitoring for the execution of a task to determine when the task has failed. Details of the failed task may be identified and used to attempt recovery from the task failure. After initiating any recovery tasks, execution of the recovery tasks may be monitored, and one or more supplementary recovery tasks may be identified and executed, or the original task may be rerun at an appropriate execution point based on the initial point of failure. Thus, when a task has failed, an iterative process may begin where various effects of the failed task are attempted to be rolled back, and depending on the success of the rollback, the initial task can be rerun at the point of failure, or further recovery tasks may be executed.

Claims

1. A system for modifying execution of scripts for a job scheduler using deontic logic, comprising:

a job scheduler configured to execute a schedule that includes at least one task associated with a goal having a first deontic modality associated therewith and create one or more event logs that include information associated with executing the at least one task in the schedule; and

a rollback and rerun supervisor communicatively coupled to the job scheduler, wherein the rollback and rerun supervisor is configured to;

determine whether the at least one task has succeeded or failed based on whether the information in the one or more event logs indicates that the goal associated with the at least one task has been achieved in a manner consistent with the first deontic modality;

determine a second deontic modality associated with recovering from failure of the at least one task if the information in the one or more event logs indicates that the at least one task has failed to achieve the goal in a manner consistent with the first deontic modality; and

load a recovery script to the job scheduler based on the second deontic modality associated with recovering from failure of the at least one task, wherein the recovery script loaded to the job scheduler includes a task description designed to achieve the goal associated with the at least one task in a manner consistent with the first deontic modality.

2. The system of claim 1, wherein to determine whether the at least one task has succeeded or failed, the rollback and rerun supervisor is further configured to:

determine that the at least one task has succeeded if the first deontic modality comprises an Obliged modality and the information in the one or more event logs indicates that the goal has been achieved; and

determine that the at least one task has failed if the first deontic modality comprises the Obliged modality and the information in the one or more event logs indicates that goal has not been achieved.

3. The system of claim 2, wherein the task description includes one or more tasks designed to achieve the goal if the first deontic modality comprises the Obliged modality and the information in the one or more event logs indicates that the goal has not been achieved.

4. The system of claim 1, wherein to determine whether the at least one task has succeeded or failed, the rollback and rerun supervisor is further configured to:

determine that the at least one task has succeeded if the first deontic modality comprises an Interdicted modality and the information in the one or more event logs indicates that the goal has not been achieved; and

determine that the at least one task has failed if the first deontic modality comprises the Interdicted modality and the information in the one or more event logs indicates that the goal has been achieved.

5. The system of claim 4, wherein the task description includes one or more tasks designed to rollback the goal if the first deontic modality comprises the Interdicted modality and the information in the one or more event logs indicates that the goal has been achieved.

6. The system of claim 1, wherein to determine whether the at least one task has succeeded or failed, the rollback and rerun supervisor is further configured to determine that the at least one task has succeeded if the first deontic modality comprises a Permitted modality whether or not the information in the one or more event logs indicates that the goal has been achieved.

7. The system of claim 1, wherein the second deontic modality associated with recovering from failure of the at least one task includes a rollback modality.

8. The system of claim 1, wherein the second deontic modality associated with recovering from failure of the at least one task includes a rerun modality.

9. The system of claim 7, wherein the rollback modality indicates whether rolling back the failure of the at least one task is Obliged, Permitted, or Interdicted.

10. The system of claim 8, wherein the rerun modality indicates whether rerunning the at least one task is Obliged, Permitted, or Interdicted.

11. The system of claim 1, wherein the rollback and rerun supervisor is further configured to cancel one or more upstream goals associated with the schedule if the information in the one or more event logs indicates that the at least one task has completed or that the at least one task has been cancelled.

12. A method for executing a job schedule, comprising:

defining a schedule to execute a plurality of tasks;

executing a first one of the plurality of tasks according to the defined schedule, wherein the first executed task includes a first goal having an Interdicted deontic modality;

generating an execution event log that includes information to indicate whether the first goal having the Interdicted deontic modality has been achieved based on an execution status associated with the first executed task; and

attempting to identify a first recovery schedule to rollback the first executed task if the information in the execution event log indicates that the first goal having the Interdicted deontic modality has been achieved, wherein the first recovery schedule includes a first recovery task associated with a goal having an Obliged deontic modality that requires a rollback to unachieve the first goal having the Interdicted deontic modality.

13. The method of claim 12, further comprising:

executing the first recovery task according to the first recovery schedule;

generating a recovery event log that includes information to indicate whether the goal having the Obliged deontic modality has achieved the required rollback to unachieved the first goal having the Interdicted deontic modality based on an execution status associated with the first recovery task; and

determining whether the first recovery schedule has succeeded or failed based on whether the information in the recovery event log indicates that the first recovery task has achieved the required rollback.

14. The method of claim 13, wherein the first recovery schedule further includes at least one rerun job to rerun the first executed task associated with the unachieved first goal having the Interdicted deontic modality and the method further comprises executing the at least one rerun job if the information in the recovery event log indicates that the first recovery schedule task has achieved the required rollback.

15. The method of claim 12, further comprising:

executing a second one of the plurality of tasks according to the defined schedule, wherein the second executed task includes a second goal having an Obliged deontic modality and the generated execution event log further includes information to indicate whether the second goal having the Obliged deontic modality has been achieved based on an execution status associated with the second executed task; and

attempting to identify a second recovery schedule to achieve the second goal associated with the second executed task if the information in the execution event log indicates that the second goal having the Obliged deontic modality has not been achieved, wherein the second recovery schedule includes a second recovery task to achieve the second goal having the Obliged deontic modality.

16. The method of claim 15, further comprising resuming execution associated with the plurality of tasks according to the defined schedule if the first recovery schedule has successfully unachieved the first goal having the Interdicted deontic modality and the second recovery scheduled has successfully achieved the second goal having the Obliged deontic modality.

17. A method for modifying execution of scripts for a job scheduler using deontic logic, comprising:

using a job scheduler to execute a schedule that includes a composite task having multiple dependent sub-tasks and one or more composite operators to define a manner to execute the multiple dependent sub-tasks, wherein at least one of the multiple dependent sub-tasks includes a goal having a first deontic modality associated therewith;

creating, via the job scheduler, an event log that includes information associated with executing the multiple dependent sub-tasks associated with the composite task;

determining, at a rollback and rerun supervisor, whether the composite task has succeeded or failed based on whether the information in the event log indicates that the goal associated with the at least one dependent sub-task has been achieved in a manner consistent with the first deontic modality;

determining, at the rollback and rerun supervisor, a second deontic modality to recover from failure associated with the composite task if the information in the event log indicates that the at least one dependent sub-task has failed to achieve the goal associated therewith in a manner consistent with the first deontic modality; and

loading, via the rollback and rerun supervisor, a recovery script to the job scheduler based on the second deontic modality to recover from the failure associated with the composite task, wherein the loaded recovery script includes a task description designed to achieve the goal associated with the at least one dependent sub-task in a manner consistent with the first deontic modality.

18. The method of claim 17, wherein the one or more composite operators include a sequence operator that defines a predetermined order in which to execute the multiple dependent sub-tasks.

19. The method of claim 17, wherein the one or more composite operators include a parallel operator that permits the multiple dependent sub-tasks to be executed in parallel to one another.

20. The method of claim 17, wherein the one or more composite operators include a fork operator that only permits a first one of the multiple dependent sub-tasks that provides an affirmative signal to be executed.

21. The method of claim 17, wherein the one or more composite operators include a select operator that provides a predicate to evaluate the multiple dependent sub-tasks in a predetermined order and select a first one of the multiple dependent sub-tasks that evaluates the predicate to true to be executed.

22. The method of claim 17, wherein the one or more composite operators include a choice operator that defines a first one of the multiple dependent sub-tasks to initiate a predetermined event associated with the first dependent sub-task to control initiating subsequent ones of the multiple dependent sub-tasks.

23. A method for modifying execution of scripts for a job scheduler using deontic logic, comprising:

using a job scheduler to execute a schedule that includes at least one task associated with a goal having a first deontic modality associated therewith, wherein the job scheduler creates one or more event logs that include information associated with executing the at least one task in the schedule;

determining, at a rollback and rerun supervisor communicatively coupled to the job scheduler, whether the at least one task has succeeded or failed based on whether the information in the one or more event logs indicates that the goal associated with the at least one task has been achieved in a manner consistent with the first deontic modality;

determining, at the rollback and rerun supervisor, a second deontic modality associated with recovering from failure of the at least one task if the rollback and rerun supervisor determines that the information in the one or more event logs indicates that the at least one task has failed to achieve the goal in a manner consistent with the first deontic modality; and

loading, via the rollback and rerun supervisor, a recovery script to the job scheduler based on the second deontic modality associated with recovering from failure of the at least one task, wherein the recovery script loaded to the job scheduler includes a task description designed to achieve the goal associated with the at least one task in a manner consistent with the first deontic modality.

24. The method of claim 23, wherein determining whether the at least one task has succeeded or failed includes:

determining that the at least one task has succeeded if the first deontic modality comprises an Obliged modality and the information in the one or more event logs indicates that the goal has been achieved; and

determining that the at least one task has failed if the first deontic modality comprises the Obliged modality and the information in the one or more event logs indicates that the goal has not been achieved.

25. The method of claim 24, wherein the task description includes one or more tasks designed to achieve the goal if the first deontic modality comprises the Obliged modality and the information in the one or more event logs indicates that the goal has not been achieved.

26. The method of claim 23, wherein determining whether the at least one task has succeeded or failed includes:

determining that the at least one task has succeeded if the first deontic modality comprises an Interdicted modality and the information in the one or more event logs indicates that the goal has not been achieved; and

determining that the at least one task has failed if the first deontic modality comprises the Interdicted modality and the information in the one or more event logs indicates that the goal has been achieved.

27. The method of claim 26, wherein the task description includes one or more tasks designed to rollback the goal if the first deontic modality comprises the Interdicted modality and the information in the one or more event logs indicates that the goal has been achieved.

28. The method of claim 23, wherein determining whether the at least one task has succeeded or failed includes determining that the at least one task has succeeded if the first deontic modality comprises a Permitted modality whether or not the information in the one or more event logs indicates that the goal has been achieved.

29. The method of claim 23, wherein the second deontic modality associated with recovering from failure of the at least one task includes a rollback modality.

30. The method of claim 29, wherein the rollback modality indicates whether rolling back the failure of the at least one task is Obliged, Permitted, or Interdicted.

31. The method of claim 23, wherein the second deontic modality associated with recovering from failure of the at least one task includes a rerun modality.

32. The method of claim 31, wherein the rerun modality indicates whether rerunning the at least one task is Obliged, Permitted, or Interdicted.

33. The method of claim 23, further comprising cancelling, by the rollback and rerun supervisor, one or more upstream goals associated with the schedule if the information in the one or more event logs indicates that the at least one task has completed or that the at least one task has been cancelled.

34. A system for executing a job schedule, wherein the system comprises one or more processors configured to:

define a schedule to execute a plurality of tasks;

execute a first one of the plurality of tasks according to the defined schedule, wherein the first executed task includes a first goal having an Interdicted deontic modality;

generate an execution event log that includes information to indicate whether the first goal having the Interdicted deontic modality has been achieved based on an execution status associated with the first executed task; and

attempt to identify a first recovery schedule to rollback the first executed task if the information in the execution event log indicates that the first goal having the Interdicted deontic modality has been achieved, wherein the first recovery schedule includes a first recovery task associated with a goal having an Obliged deontic modality that requires a rollback to unachieve the first goal having the Interdicted deontic modality.

35. The system of claim 34, wherein the one or more processors are further configured to:

execute the first recovery task according to the first recovery schedule;

generate a recovery event log that includes information to indicate whether the goal having the Obliged deontic modality has achieved the required rollback to unachieve the first goal having the Interdicted deontic modality based on an execution status associated with the first recovery task; and

determine whether the first recovery schedule has succeeded or failed based on whether the information in the recovery event log indicates that the first recovery task has achieved the required rollback.

36. The system of claim 35, wherein the first recovery schedule further includes at least one rerun job to rerun the first executed task associated with the unachieved first goal having the Interdicted deontic modality and the one or more processors are further configured to execute the at least one rerun job if the information in the recovery event log indicates that the first recovery task has achieved the required rollback.

37. The system of claim 34, wherein the one or more processors are further configured to:

execute a second one of the plurality of tasks according to the defined schedule, wherein the second executed task includes a second goal having an Obliged deontic modality and the generated execution event log further includes information to indicate whether the second goal having the Obliged deontic modality has been achieved based on an execution status associated with the second executed task; and

attempt to identify a second recovery schedule to achieve the second goal associated with the second executed task if the information in the execution event log indicates that the second goal having the Obliged deontic modality has not been achieved, wherein the second recovery schedule includes a second recovery task to achieve the second goal having the Obliged deontic modality.

38. The system of claim 37, wherein the one or more processors are further configured to resume execution associated with the plurality of tasks according to the defined schedule if the first recovery schedule has successfully unachieved the first goal having the Interdicted deontic modality and the second recovery scheduled has successfully achieved the second goal having the Obliged deontic modality.

39. A system for modifying execution of scripts for a job scheduler using deontic logic, wherein the system comprises one or more processors configured to:

execute a schedule that includes a composite task having multiple dependent sub-tasks and one or more composite operators to define a manner to execute the multiple dependent sub-tasks, wherein at least one of the multiple dependent sub-tasks includes a goal having a first deontic modality associated therewith;

create an event log that includes information associated with executing the multiple dependent sub-tasks associated with the composite task;

determine whether the composite task has succeeded or failed based on whether the information in the event log indicates that the goal associated with the at least one dependent sub-task has been achieved in a manner consistent with the first deontic modality;

determine a second deontic modality to recover from failure associated with the composite task if the information in the event log indicates that the at least one dependent sub-task has failed to achieve the goal associated therewith in a manner consistent with the first deontic modality; and

load a recovery script based on the second deontic modality to recover from the failure associated with the composite task, wherein the loaded recovery script includes a task description designed to achieve the goal associated with the at least one dependent sub-task in a manner consistent with the first deontic modality.

40. The system of claim 39, wherein the one or more composite operators include a sequence operator that defines a predetermined order in which to execute the multiple dependent sub-tasks.

41. The system of claim 39, wherein the one or more composite operators include a parallel operator that permits the multiple dependent sub-tasks to be executed in parallel to one another.

42. The system of claim 39, wherein the one or more composite operators include a fork operator that only permits a first one of the multiple dependent sub-tasks to provide an affirmative signal to be executed.

43. The system of claim 39, wherein the one or more composite operators include a select operator that provides a predicate to evaluate the multiple dependent sub-tasks in a predetermined order and select a first one of the multiple dependent sub-tasks that evaluates the predicate to true to be executed.

44. The system of claim 39, wherein the one or more composite operators include a choice operator that defines a first one of the multiple dependent sub-tasks to initiate and a predetermined event associated with the first dependent sub-task to control initiating, subsequent ones of the multiple dependent sub-tasks.