Systems and methods for monitoring and acting on logged system messages

• US 8,868,983 B1
• Filed: 06/18/2012
• Issued: 10/21/2014
• Est. Priority Date: 03/14/2008
• Status: Active Grant

• Current Assignee: United Services Automobile Association
• Original Assignee: United Services Automobile Association
• Inventors: Leverich, Josiah J., Hobrecht, Max Jr.
• Primary Examiner(s): BONZO, BRYCE P
• Application Number:
  US13/525,723
• Time in Patent Office: 855 Days
• Field of Search: 714/47
• US Class Current: 714/47.1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/3409 for performance assessment

Abstract

A computer operations center that has a number of computing systems, where each computing system issues system messages regarding such computing system. Each issued system message from each computing system is stored in a message logging database. The system messages stored in the message logging database are monitored for predetermined patterns, and upon detecting a particular pattern in the system messages, a predetermined action corresponding to the detected pattern is taken. The detected pattern anticipates an occurrence of an unwanted system event, and the corresponding action is intended to prevent or at least minimize the unwanted system event.

Claims

1. A method with regard to a computer operations center having a plurality of computing systems, the plurality of computing systems issuing system messages that are stored in a message logging database, the method comprising:

monitoring the system messages stored in the message logging database for a predetermined pattern in the system messages;

detecting a particular pattern in the system messages;

querying the message logging database for relevant messages;

computing a statistic based on the relevant messages;

obtaining, from the detected pattern, a first threshold and a first action to be taken if the first threshold is surpassed by the statistic, wherein the detected pattern further includes a second threshold, and a third threshold; and

acting in response to the detected pattern by taking a predetermined action corresponding to the detected pattern that anticipates an occurrence of an unwanted system event, the predetermined action intended to prevent or at least minimize the unwanted system event, wherein the predetermined action corresponding to the detected pattern includes the first action to be taken if the first threshold is satisfied, a second corresponding action to be taken if the second threshold is satisfied, and a third corresponding action to be taken if the third threshold is satisfied, and wherein the first corresponding action, the second corresponding action, and the third corresponding action are each different from each other.

2. The method of claim 1 further comprising:

detecting a second pattern in the system messages stored in the message logging database;

comparing the second pattern to the predetermined pattern; and

determining that the second pattern does not match the predetermined pattern, and in response to this determination;

determining that the second pattern anticipates the occurrence of the unwanted system event; and

defining, in a pattern/action database and in response to the determination that the second pattern anticipates the unwanted system event, the satisfied pattern, thresholds that correspond to the second pattern, and actions that correspond to the second pattern.

3. The method of claim 1, wherein the predetermined pattern comprises a plurality of predetermined patterns, and wherein each of the plurality of predetermined patterns is defined to include a particular message or type of message to be monitored and a rate of occurrence thereof.

4. The method of claim 1, wherein the first and second corresponding actions are each selected from a group including issuing a notification, initiating a maintenance operation, and shutting down a corresponding one of the computing systems.

5. The method of claim 1 wherein a statistics generator stores the statistic in a statistics database and a MAAS retrieves the statistic from the statistics database.

6. The method of claim 1 wherein a statistics generator generates the statistic by querying the message logging database for relevant messages from a window representing a recent time period.

7. A non-transitory computer-readable medium having stored thereon computer-executable instructions implementing a method with regard to a computer operations center having a plurality of computing systems, the plurality of computing systems issuing system messages that are stored in a message logging database, the instructions causing a computer processor to:

monitor the system messages stored in the message logging database for a predetermined pattern in the system messages;

detect a particular pattern in the system messages;

query the message logging database for relevant messages;

compute a statistic based on the relevant messages;

obtain, from the detected pattern, a first threshold and a first action to be taken if the first threshold is surpassed by the statistic, wherein the detected pattern further includes a second threshold, and a third threshold; and

act in response to the detected pattern by taking a predetermined action corresponding to the detected pattern that anticipates an occurrence of an unwanted system event, the predetermined action intended to prevent or at least minimize the unwanted system event, wherein the predetermined action corresponding to the detected pattern includes the first action to be taken if the first threshold is satisfied, a second corresponding action to be taken if the second threshold is satisfied, and a third corresponding action to be taken if the third threshold is satisfied, and wherein the first corresponding action, the second corresponding action, and the third corresponding action are each different from each other.

8. The computer-readable medium of claim 7, the instructions further causing the processor to:

detect a second pattern in the system messages stored in the message logging database;

compare the second pattern to the predetermined pattern; and

determine that the second pattern does not match the predetermined pattern, and in response to this determination;

determine that the second pattern anticipates the occurrence of the unwanted system event; and

define, in a pattern/action database and in response to the determination that the second pattern anticipates the unwanted system event, the satisfied pattern, thresholds that correspond to the second pattern, and actions that correspond to the second pattern.

9. The computer-readable medium of claim 7, wherein the predetermined pattern comprises a plurality of predetermined patterns, and wherein each of the plurality of predetermined patterns is defined to include a particular message or type of message to be monitored and a rate of occurrence thereof.

10. The computer-readable medium of claim 7, wherein the first and second corresponding actions are each selected from a group including issuing a notification, initiating a maintenance operation, and shutting down a corresponding one of the computing systems.

11. The computer-readable medium of claim 7, wherein a statistics generator generates the statistic by querying the message logging database for relevant messages from a window representing a recent time period.

12. The computer-readable medium of claim 7, wherein a statistics generator stores the statistic in a statistics database and a MAAS retrieves the statistic from the statistics database.

13. A system with regard to a computer operations center having a plurality of computing systems, the plurality of computing systems issuing system messages that are stored in a message logging database, the system comprising:

a sub-system that monitors the system messages stored in the message logging database for a predetermined pattern in the system messages;

a sub-system that detects a particular pattern in the system messages;

a sub-system that queries the message logging database for relevant messages;

a sub-system that computes a statistic based on the relevant messages;

a sub-system that obtains, from the detected pattern, a first threshold and a first action to be taken if the first threshold is surpassed by the statistic, wherein the detected pattern further includes a second threshold, and a third threshold; and

a sub-system that acts in response to the detected pattern by taking a predetermined action corresponding to the detected pattern that anticipates an occurrence of an unwanted system event, the predetermined action intended to prevent or at least minimize the unwanted system event, wherein the predetermined action corresponding to the detected pattern includes the first action to be taken if the first threshold is satisfied, a second corresponding action to be taken if the second threshold is satisfied, and a third corresponding action to be taken if the third threshold is satisfied, and wherein the first corresponding action, the second corresponding action, and the third corresponding action are each different from each other.

14. The system of claim 13 further comprising:

a sub-system that detects a second pattern in the system messages stored in the message logging database;

a sub-system that compares the second pattern to the predetermined pattern; and

a sub-system that determines that the second pattern does not match the predetermined pattern, and in response to this determination;

determine that the second pattern anticipates the occurrence of the unwanted system event; and

define, in a pattern/action database and in response to the determination that the second pattern anticipates the unwanted system event, the satisfied pattern, thresholds that correspond to the second pattern, and actions that correspond to the second pattern.

15. The system of claim 13, wherein the predetermined pattern comprises a plurality of predetermined patterns, and wherein each of the plurality of predetermined patterns is defined to include a particular message or type of message to be monitored and a rate of occurrence thereof.

16. The system of claim 13, wherein the first and second corresponding actions are each selected from a group including issuing a notification, initiating a maintenance operation, and shutting down a corresponding one of the computing systems.

17. The system of claim 13, wherein a statistics generator stores the statistic in a statistics database and a MAAS retrieves the statistic from the statistics database.