ADDING SCALABILITY AND FAULT TOLERANCE TO GENERIC FINITE STATE MACHINE FRAMEWORKS FOR USE IN AUTOMATED INCIDENT MANAGEMENT OF CLOUD COMPUTING INFRASTRUCTURES

• US 20120151272A1
• Filed: 12/09/2010
• Published: 06/14/2012
• Est. Priority Date: 12/09/2010
• Status: Active Grant

• Current Assignee: Globalfoundries US Incorporated (GlobalFoundries, Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Behrendt, Michael M., Hosn, Rafah A., Mahindru, Ruchi, Sarkar, Soumitra, Viswanathan, Mahesh, Vogl, Norbert G., Ramasamy, Harigovind V.
• Granted Patent: US 8,560,887 B2
• Time in Patent Office: Days
• US Class Current: 714/39
• CPC Class Codes: G06F 11/203 using migration
• CPC Class Codes: G06F 11/2041 with more than one idle spare processing component
• CPC Class Codes: G06F 11/2043 where the redundant components share a common memory address space
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications

Abstract

A scalable and fault tolerant finite state machine engine, for example, for use in an automated incident management system, logs or records data in persistent storage at different points or levels during various internal processing of an event associated with an information technology element, and action taken associated with the event, by executing a finite state machine instance that encodes policies for handling incidents on such types of information technology elements. In the event that the finite state machine engine is shutdown during processing, the finite state machine engine is able to pick up from where it left off when it was shutdown, for each abnormally terminated finite state machine instance, by using the data logged in the persistent storage and determining a point of processing from where it should continue its execution.

Claims

1. A system for executing finite state machines in a scalable and fault tolerant manner, comprising:

one or more processors;

persistent storage for storing data associated with handling of an event associated with an information technology element;

a finite state machine engine operable to receive the event and execute a finite state machine instance representing the information technology element, and operable to log in the persistent storage, a plurality of internal actions associated with processing of the event, wherein if the finite state machine engine is stopped while in execution, the finite state machine engine continues to process the event based on the persistent data logged in persistent storage indexed by the finite state machine instance to complete the handling of the event, after the finite state machine engine is restarted.

2. The system of claim 1, wherein the finite state machine engine logs data at different log levels in persistent storage during the processing of the event, and wherein the finite state machine engine determines, after it is restarted, if the finite state machine instance terminated abnormally, and if so, locates the correct point in the instance from which to continue execution for the continuing the processing of the event, by using the data logged at different log levels.

3. The system of claim 1, wherein the finite state machine engine receives the event from an external module and queues the event into an event queue, the event queue being stored in the persistent storage.

4. The system of claim 1, wherein the finite state machine engine schedules a thread to handle the event and logs data associated with the scheduled thread handling the event, the thread invoking an instance of the finite state machine representing the information technology element, wherein the instance of finite state machine executes state transitions associated with the information technology element until blocking and returns control to the finite state machine engine, at which time current state information of the finite state machine instance is logged in the persistent storage and the finite state machine instance is deleted.

5. The system of claim 4, wherein the instance of the finite state machine is operable to queue a workflow with an external workflow module during its execution.

6. The system of claim 5, wherein the instance of the finite state machine assigns a unique identifier to the workflow, logs data indicating the workflow is to be submitted, and after submitting the workflow, logs data indicating the workflow was submitted and also logs data identifying the workflow unique identifier as being valid.

7. The system of claim 6, wherein the workflow performs a validation step with the finite state machine engine presenting the workflow unique identifier, and the finite state machine, if resubmitting the workflow during log replay after an abnormal termination, invalidates the previously assigned unique identifier in the workflow identifier log, and assigns a new unique identifier to the resubmitted workflow, to prevent nonidempotent workflows from being reexecuted during log replay.

8. The system of claim 1, wherein the persistent data store includes an event queue operable to store a plurality of events respectively associated with an information technology element, a finite state machine states table operable to store information associated with a plurality of states respectively associated with a plurality of finite state machines each associated with a type of information technology element, a workflow unique identifier table operable to store a plurality of scheduled workflows, a timer table operable to store timers scheduled by running finite state machine instances, and an event-action history table operable to store information associated with a plurality of events and associated corrective actions taken during executions of finite state machine instances.

9. The system of claim 1, wherein the finite state machine engine logs data at different log levels in the persistent storage during the processing of the event, and wherein the finite state machine engine determines the correct point in the processing of the event from which to continue execution after the finite state machine engine is restarted by using the data logged at different log levels, wherein the different log levels are determined based on whether one or more different entries have been updated in one or more of:

a finite state machine states table operable to store information associated with a plurality of states respectively associated with a plurality of finite state machines each associated with a type of information technology element, a workflow unique identifier table operable to store a plurality of scheduled workflows, a timer table operable to store timers scheduled by running finite state machine instances, and an event-action history table operable to store information associated with a plurality of events and associated corrective actions and status of whether a workflow is being submitted or has been submitted.

10. The system of claim 9, wherein multiple finite state machine engines are configured in an active-active configuration using distributed state sharing technology, and a shared state that includes information that is logged by each finite state machine engine.

11. A method for implementing scalable, fault-tolerant finite state machines, comprising:

receiving an event associated with an information technology element;

identifying an entry associated with the information technology element in a finite state machine states table stored in persistent storage;

updating the entry as locked in the finite state machine states table and storing the updated finite state machine states table in the persistent storage;

creating a finite state machine instance to process the event and initializing it with information from the finite state machine states table;

invoking an execute operation of the finite state machine instance, wherein the finite state machine instance executes state transitions associated with the information technology element, starting from the point where it finished execution when it was last invoked;

logging data in the persistent storage indicating that a workflow is to be scheduled before submitting the workflow;

logging data in the persistent storage indicating that the workflow is submitted after scheduling the workflow; and

after the finite state machine instance finishes, updating the entry as unlocked, and also storing current state information to be used in the next execution, in the finite state machine states table.

12. The method of claim 11, wherein after restart of the finite state machine engine, for each abnormally terminated finite state machine instance, the finite state machine engine determines a point in processing of the event based on log data stored in the persistent storage and continues execution of the finite state machine instance to process the event from the determined point.

13. The method of claim 12, wherein after the restart of the finite state machine engine, the finite state machine engine selects one or more entries in the finite machine states table having locked status, and continues to process one or more events specified in the selected one or more entries from the determined point in the corresponding finite state machine instance.

14. The method of claim 13, wherein the determined point is creating a finite state machine instance to process the event from the original state logged in the finite state machine states table, if no entries are logged associated with event action history.

15. The method of claim 13, wherein the determined point is creating a finite state machine instance to process the event from the state reserved for scheduling the workflow, if log data in the persistent storage indicates that a workflow associated with the event is to be submitted.

16. The method of claim 13, wherein the determined point is updating the entry as unlocked and the current state, in the finite state machine states table, if the data in the persistent storage indicates that the workflow was submitted.

17. The method of claim 11, wherein the step of receiving includes an event queue polling thread polling an event queue in the persistent storage.

18. The method of claim 11, wherein after the updating step, a thread is scheduled for creating the finite state machine instance, initializing parameters required for its execution, and invoking the finite state machine instance, and wherein after the finite state machine instance finishes it exits and returns control to the thread.

19. A method for executing finite state machines in a scalable and fault-tolerant manner, comprising:

an event queue polling thread polling an event queue in persistent storage;

in response to receiving an event in the event queue, the event queue polling thread associating the event with a finite state machine instance representing an information technology infrastructure element based on information stored in a finite state machine states table;

the event queue polling thread inserting a work item including an identifier associated with the information technology element, the type of element, the current state of the finite state machine and the event identifier and event name, in an in-memory thread pool queue;

the thread pool further allocating a thread to run the finite state machine;

the thread creating a finite state machine instance associated with the information technology element, initializing it with values set in the work item, and executing the finite state machine instance until the finite state machine quiesces;

the finite state machine instance retrieving data from an event-action history table to determine one or more corrective actions to be taken associated with the event;

the finite state machine instance logging the one or more corrective actions to be taken and a workflow unique identifier associated with the one or more corrective actions to be taken in a workflow unique identifier table;

the finite state machine instance scheduling the one or more corrective actions with a workflow system module for executing corrective actions,

the finite state machine performing a validation protocol with each scheduled workflow before permitting it to continue execution when scheduled,

wherein in response to a processor executing the steps of the method being shutdown and restarted, the finite state machine can continue to process the event from a point of processing before the shutdown based on logs stored in the persistent storage during the steps.

20. A computer readable storage medium storing a program of instructions executable by a machine to perform a method for scalable, fault-tolerant finite state machine, comprising:

receiving an event associated with an information technology element;

identifying an entry associated with the information technology element in a finite state machine states table stored in persistent storage;

updating the entry as locked in the finite state machine states table and storing the updated finite state machine states table in the persistent storage;

creating a finite state machine instance to process the event and initializing it with information from the finite state machine states table;

invoking an execute operation of the finite state machine instance, wherein the finite state machine instance executes state transitions associated with the information technology element, starting from the point where it finished execution when it was last invoked;

logging data in the persistent storage indicating that a workflow is to be scheduled before submitting the workflow;

logging data in the persistent storage indicating that the workflow is submitted after scheduling the workflow; and

after the finite state machine instance finishes, updating the entry as unlocked, and also storing current state information to be used in the next execution, in the finite state machine states table.

21. The computer storage medium of claim 20, wherein after restart of the finite state machine, the finite state machine determines a point in processing of the event based on data stored in the persistent storage and continues to process the event from the determined point.

22. The computer storage medium of claim 21, wherein after the restart of the finite state machine, the finite state machine selects one or more entries in the finite machine states table having locked status, and continues to process one or more events specified in the selected one or more entries from the determined point.

23. The computer storage medium of claim 22, wherein the determined point is creating a finite state machine instance to process the event, if no entries are logged associated with event action history.

24. The computer storage medium of claim 22, wherein the determined point is scheduling the workflow if data in the persistent storage indicates that a workflow associated with the event is to be submitted.

25. The computer storage medium of claim 22, wherein the determined point is updating the entry as unlocked and a current state, in the finite state machine states table, if the data in the persistent storage indicates that the workflow was submitted.