Systems and methods for monitoring and acting on logged system messages

• US 8,205,122 B1
• Filed: 03/14/2008
• Issued: 06/19/2012
• Est. Priority Date: 03/14/2008
• Status: Active Grant

• Current Assignee: United Services Automobile Association
• Original Assignee: United Services Automobile Association
• Inventors: Leverich, Josiah J., Hobrecht, Max Jr.
• Primary Examiner(s): Bonzo, Bryce
• Application Number:
  US12/048,863
• Time in Patent Office: 1,558 Days
• Field of Search: 714/47
• US Class Current: 714/47.1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/3409 for performance assessment

Abstract

A computer operations center that has a number of computing systems, where each computing system issues system messages regarding such computing system. Each issued system message from each computing system is stored in a message logging database. The system messages stored in the message logging database are monitored for predetermined patterns, and upon detecting a particular pattern in the system messages, a predetermined action corresponding to the detected pattern is taken. The detected pattern anticipates an occurrence of an unwanted system event, and the corresponding action is intended to prevent or at least minimize the unwanted system event.

Claims

1. A method with regard to a computer operations center having a number of computing systems, each computing system issuing system messages regarding such computing system, each issued system message from each computing system being stored in a message logging database, the method comprising:

monitoring the system messages stored in the message logging database for predetermined patterns in the system messages, wherein the system messages include error messages, maintenance messages, and status messages;

detecting a particular pattern in the system messages, wherein each pattern is defined to include at least a first threshold, a second threshold, and a third threshold and wherein the error messages, maintenance messages, and status messages forming the particular pattern are analyzed to anticipate one or more bad system events;

acting in response to the detected pattern by taking a predetermined action corresponding to the detected pattern, the detected pattern anticipating an occurrence of an unwanted system event, the predetermined action intended to prevent or at least minimize the unwanted system event, wherein the predetermined action corresponding to the detected pattern includes a first corresponding action to be taken if the first threshold is surpassed, a second corresponding action to be taken if the second threshold is surpassed, and a third corresponding action to be taken if the third threshold is surpassed, and wherein the first corresponding action and the second corresponding action are different from each other and the third corresponding action includes shutting down a corresponding one of the computing systems;

monitoring the system messages stored in the message logging database for non-predetermined patterns in the system messages;

receiving a determination to whether the non-predetermined pattern anticipates the occurrence of the unwanted system event; and

defining, in response to the determination that the non-predetermined pattern anticipates the unwanted system event, the non-predetermined pattern, corresponding thresholds, and corresponding actions in a patter/action database.

2. The method of claim 1 wherein a monitoring and acting system (MAAS) is provided to implement the method.

3. The method of claim 1 wherein each pattern is defined to include a particular message or type of message to be monitored and a rate of occurrence thereof.

4. The method of claim 1 wherein the first and second corresponding actions are selected from a group including issuing a notification, initiating a maintenance operation, and shutting down a corresponding one of the computing systems.

5. The method of claim 1 wherein a statistics generator generates a needed statistic by:

determining from the particular pattern the needed statistic;

querying the message logging database for relevant messages; and

computing the needed statistic based on the relevant messages, and wherein a monitoring and acting system (MAAS) employs the generated statistic by;

obtaining from the particular pattern a threshold value and a corresponding action to be taken if the threshold value is surpassed by the generated statistic;

comparing the generated statistic to the obtained threshold value to determine whether the threshold value is surpassed by the generated statistic; and

upon determining that the threshold value is surpassed by the generated statistic, performing the corresponding action.

6. The method of claim 5 wherein the statistics generator stores the generated statistic in a statistics database and the MAAS retrieves the generated statistic from the statistics database.

7. The method of claim 5 wherein the statistics generator generates the needed statistic by querying the message logging database for relevant messages from a window representing a recent time period, and ignoring older messages.

8. A computer-readable medium having stored thereon computer-executable instructions implementing a method with regard to a computer operations center having a number of computing systems, each computing system issuing system messages regarding such computing system, each issued system message from each computing system being stored in a message logging database, the method comprising:

monitoring the system messages stored in the message logging database for predetermined patterns in the system messages, wherein the system messages include error messages, maintenance messages, and status messages;

detecting a particular pattern in the system messages, wherein each pattern is defined to include at least a first threshold, a second threshold, and a third threshold and wherein the error messages, maintenance messages, and status messages forming the particular pattern are analyzed to anticipate one or more bad system events;

acting in response to the detected pattern by taking a predetermined action corresponding to the detected pattern, the detected pattern anticipating an occurrence of an unwanted system event, the predetermined action intended to prevent or at least minimize the unwanted system event, wherein the predetermined action corresponding to the detected pattern includes a first corresponding action to be taken if the first threshold is surpassed, a second corresponding action to be taken if the second threshold is surpassed, and a third corresponding action to be taken if the third threshold is surpassed, and wherein the first corresponding action and the second corresponding action are different from each other and the third corresponding action includes shutting down a corresponding one of the computing systems;

monitoring the system messages stored in the message logging database for non-predetermined patterns in the system messages;

receiving a determination whether the non-predetermined pattern anticipates the occurrence of the unwanted system event; and

defining, in response to the determination that the non-predetermined pattern anticipates the unwanted system event, the non-predetermined pattern, corresponding thresholds, and corresponding actions in a patter/action database.

9. The medium of claim 8 wherein a monitoring and acting system (MAAS) is provided to implement the method.

10. The medium of claim 8 wherein each pattern is defined to include a particular message or type of message to be monitored and a rate of occurrence thereof.

11. The medium of claim 8 wherein the first and second corresponding actions are selected from a group including issuing a notification, initiating a maintenance operation, and shutting down a corresponding one of the computing systems.

12. The medium of claim 8 wherein a statistics generator generates a needed statistic by:

determining from the particular pattern the needed statistic;

querying the message logging database for relevant messages; and

computing the needed statistic based on the relevant messages,

and wherein a monitoring and acting system (MAAS) employs the generated statistic by;

obtaining from the particular pattern a threshold value and a corresponding action to be taken if the threshold value is surpassed by the generated statistic;

comparing the generated statistic to the obtained threshold value to determine whether the threshold value is surpassed by the generated statistic; and

upon determining that the threshold value is surpassed by the generated statistic, performing the corresponding action.

13. The medium of claim 12 wherein the statistics generator stores the generated statistic in a statistics database and the MAAS retrieves the generated statistic from the statistics database.

14. The medium of claim 12 wherein the statistics generator generates the needed statistic by querying the message logging database for relevant messages from a window representing a recent time period, and ignoring older messages.

15. A system with regard to a computer operations center having a number of computing systems, each computing system issuing system messages regarding such computing system, each issued system message from each computing system being stored in a message logging database, the system comprising:

a sub-system that monitors the system messages stored in the message logging database for predetermined patterns in the system messages, wherein the system messages include error messages, maintenance messages, and status messages;

a sub-system that detects a particular pattern in the system messages, wherein each pattern is defined to include at least a first threshold, a second threshold, and a third threshold and wherein the error messages, maintenance messages, and status messages forming the particular pattern are analyzed to anticipate one or more bad system events;

a sub-system that acts in response to the detected pattern by taking a predetermined action corresponding to the detected pattern, the detected pattern anticipating an occurrence of an unwanted system event, the predetermined action intended to prevent or at least minimize the unwanted system event, wherein the predetermined action corresponding to the detected pattern includes a first corresponding action to be taken if the first threshold is surpassed, a second corresponding action to be taken if the second threshold is surpassed, and a third corresponding action to be taken if the third threshold is surpassed, and wherein the first corresponding action and the second corresponding action are different from each other and the third corresponding action includes shutting down a corresponding one of the computing systems;

a sub-system that monitors the system messages stored in the message logging database for non-predetermined patterns in the system messages;

a sub-system that receives a determination whether the non-predetermined pattern anticipates the occurrence of the unwanted system event; and

a sub-system that defines, in response to the determination that the non-predetermined pattern anticipates the unwanted system event, the non-predetermined pattern, corresponding thresholds, and corresponding actions in a patter/action database.

16. The system of claim 15 comprising a monitoring and acting system (MAAS).

17. The system of claim 15 wherein each pattern is defined to include a particular message or type of message to be monitored and a rate of occurrence thereof.

18. The system of claim 15 wherein the first and second corresponding actions are selected from a group including issuing a notification, initiating a maintenance operation, and shutting down a corresponding one of the computing systems.

19. The system of claim 15 comprising a statistics generator that generates a needed statistic by:

determining from the particular pattern the needed statistic;

querying the message logging database for relevant messages; and

computing the needed statistic based on the relevant messages,

and comprising a monitoring and acting system (MAAS) that employs the generated statistic by;

obtaining from the particular pattern a threshold value and a corresponding action to be taken if the threshold value is surpassed by the generated statistic;

comparing the generated statistic to the obtained threshold value to determine whether the threshold value is surpassed by the generated statistic; and

upon determining that the threshold value is surpassed by the generated statistic, performing the corresponding action.

20. The system of claim 19 wherein the statistics generator stores the generated statistic in a statistics database and the MAAS retrieves the generated statistic from the statistics database.

21. The system of claim 19 wherein the statistics generator generates the needed statistic by querying the message logging database for relevant messages from a window representing a recent time period, and ignoring older messages.