Method of assessing restart approach to minimize recovery time

• US 8,898,520 B1
• Filed: 04/19/2012
• Issued: 11/25/2014
• Est. Priority Date: 04/19/2012
• Status: Active Grant

• Current Assignee: T-Mobile Innovations LLC (Deutsche Telekom AG)
• Original Assignee: Sprint Communications Company LP (Deutsche Telekom AG)
• Inventors: Ford, Terriss L., Leonard, Mark D.
• Primary Examiner(s): GUYTON, PHILIP A
• Application Number:
  US13/451,281
• Time in Patent Office: 950 Days
• Field of Search: 714/15
• US Class Current: 714/15
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/1441 Resetting or repowering
• CPC Class Codes: G06F 2209/548 Queue
• CPC Class Codes: G06F 9/546 Message passing systems or structures, e.g. queues

Abstract

A computer implemented method is provided for message queue failure recovery. The method comprises detecting a failure in a message queue or a queue manager for the message queue, detecting a current status of each of the message queue and the queue manager, examining a maintained active log for the message queue and a message recovery log, examining usage of system resources associated with the message queue and the queue manager, and executing one of a plurality of failure recovery procedures based on the current status of the message queue and the queue manager, the active log, the message recovery log, and the usage of the system resources.

Claims

1. A system for message queue failure recovery, comprising:

a non-transitory computer readable storage medium comprising a recovery management component stored as a set of computer instructions that, when executed by a processor, cause the processor to;

detect a failure in a message queue or a queue manager for the message queue,

iteratively detect a current status of each of the message queue and the queue manager,

iteratively examine a maintained active log for the message queue and a message recovery log,

iteratively select one of a plurality of failure recovery procedures based at least on the iteratively detected current status of the message queue and the queue manager, the maintained active log, and the message recovery log, wherein the plurality of failure recovery procedures comprise;

a first procedure to restart the queue manager and reload one or more messages in the queue manager from a backup queue,

a second procedure to shut down and restart a server that hosts the message queue and the queue manager, and

a third procedure to shut down a server that hosts the message queue and the queue manager and signal a request for further investigation into the failure, and

responsive to detecting the failure, execute the currently selected one of the plurality of failure recovery procedures.

2. The system of claim 1, wherein the failure is caused by a software error, a hardware error, or a communication error.

3. The system of claim 1, wherein the current status of the message queue, the queue manager, or both is one of active, busy, idle, and non-responsive.

4. The system of claim 1, wherein the maintained active log indicates information for recovering the message queue, the queue manager, and related objects.

5. The system of claim 1, wherein the maintained active log indicates the message queue'"'"'s depth.

6. The system of claim 1, wherein the message recovery log comprises a plurality of problem identifiers (IDs) specifying identifying a plurality of message queue components with problems.

7. The system of claim 1, wherein at least one of the plurality of failure recovery procedures is executed using a configurable symbolic link that points to a selected executable routine.

8. The system of claim 1, wherein the message queue and the queue manager application are hosted in a memory component of a server that is coupled to a plurality of other servers in a data center or a network, and wherein at least some of the plurality of other servers comprise a plurality of other message queues and other queue managers in communications with the message queue and the queue manager.

9. The system of claim 1, wherein the current status of each of the message queue and the queue manager is maintained and updated in a corresponding object in a computer memory component, and wherein the active log and the message recovery log are maintained in corresponding files in a non-transitory computer readable storage medium.

10. The system of claim 1, wherein the failure is detected subsequent to the iterative detection, the iterative examination, and the iterative selection.

11. A computer implemented method for message queue failure recovery, comprising:

detecting, by a processor, a failure in a message queue or a queue manager for the message queue;

detecting, by a processor, a current status of each of the message queue and the queue manager;

examining, by a processor, a maintained active log for the message queue and a message recovery log;

examining, by a processor, usage of system resources associated with the message queue and the queue manager; and

executing one of a plurality of failure recovery procedures based on the current status of the message queue and the queue manager, the maintained active log, the message recovery log, and the usage of the system resources, wherein responsive to a difference between the maintained active log and the recovery log exceeding a determined quantity of logs, the executed failure recovery procedure shuts down a server that hosts the message queue and the queue manager and signals a request for investigation of the failure.

12. The computer implemented method of claim 11, wherein responsive to the current status of the message queue indicating a non-responsive queue, the current status of the queue manager indicating a responsive queue manager, and the maintained active log and message recovery log indicating persistent messages in the message queue, the executed failure recovery procedure at least restarts the queue manager and reloads one or more messages in the queue manager from a backup queue.

13. The computer implemented method of claim 11, wherein responsive to the current status of each of the message queue and the queue manager being non-responsive and the maintained active log and message recovery log indicating a previous restart of the message queue manager within a pre-defined time interval, the failure recovery procedure executed shuts down and restarts a server that hosts the message queue and the queue manager.

14. The computer implemented method of claim 11, wherein the current status for each of the message queue and the queue manager, the maintained active log, the message recovery log, and the usage of system resources are analyzed using a processor executing decision tree logic to select at least one of a plurality of pre-determined routines for a message queue recovery.

15. The computer implemented method of claim 11, wherein the system resources include memory and central processing unit utilization statistics.

16. A system for message queue failure recovery, comprising:

a non-transitory computer readable storage medium comprising a recovery management component stored as a set of computer instructions that, when executed by a processor, cause the processor to;

detect a failure in a message queue or a queue manager for the message queue,

iteratively detect a current status of each of the message queue and the queue manager,

iteratively examine a maintained active log for the message queue and a message recovery log,

examine status and log information for a plurality of other message queues and queue managers for the other message queues,

iteratively select at least one of a plurality of failure recovery procedures based at least on the iteratively detected current status of the message queue and the queue manager, the maintained active log, and the message recovery log,

responsive to detecting the failure, execute the currently selected at least one of the plurality of failure recovery procedures based at least on the iteratively detected current status of the message queue and the queue manager, the maintained active log, and the message recovery log, and

redistribute a plurality of messages previously assigned to the failed message queue or queue manager to the other message queues or the queue managers of the other message queues based on the respective status and log information for the message queues and queue managers for the other message queues.

17. The system of claim 16, wherein at least one of the messages previously assigned to the failed message queue or queue manager is reassigned to the same message queue after recovery.

18. The system of claim 16, wherein the messages previously assigned to the failed message queue or queue manager are reassigned to the other message queues or the queue managers of the other message queues in a manner to achieve load balancing between all active message queues and queue managers.

19. The system of claim 16, wherein the messages previously assigned to the failed message queue or queue manager are also redistributed to the other message queues or the queue managers of the other message queues based on the current status of each of the message queue and the queue manager, the maintained active log, and the message recovery log.

20. The system of claim 19, wherein the recovery management component further examines usage of system resources associated with the message queue, the queue manager, the other message queues, and the queue managers of the other message queues, and wherein the messages previously assigned to the failed message queue or queue manager are also redistributed to the other message queues or the queue managers of the other message queues based on the usage of system resources.