Systems and methods for monitoring and acting on logged system messages

• US 9,811,437 B1
• Filed: 09/10/2014
• Issued: 11/07/2017
• Est. Priority Date: 03/14/2008
• Status: Active Grant

• Current Assignee: United Services Automobile Association
• Original Assignee: United Services Automobile Association
• Inventors: Leverich, Josiah J., Hobrecht, Jr., Max
• Primary Examiner(s): Duncan, Marc
• Application Number:
  US14/482,018
• Time in Patent Office: 1,154 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/3409 for performance assessment

Abstract

A computer operations center that has a number of computing systems, where each computing system issues system messages regarding such computing system. Each issued system message from each computing system is stored in a message logging database. The system messages stored in the message logging database are monitored for predetermined patterns, and upon detecting a particular pattern in the system messages, a predetermined action corresponding to the detected pattern is taken. The detected pattern anticipates an occurrence of an unwanted system event, and the corresponding action is intended to prevent or at least minimize the unwanted system event.

Claims

1. A method comprising:

storing system messages in a message logging database, the system messages being issued by a plurality of computing systems;

monitoring the system messages stored in the message logging database for predetermined patterns in the system messages;

detecting a pattern in the system messages that indicates a potential system event, wherein the pattern comprises a frequency of a subset of the system messages, wherein the subset of the system messages indicates a potential system event;

determining multiple thresholds corresponding to the frequency of the subset of the system messages; and

acting in response to the pattern by taking a predetermined action corresponding to the pattern, the predetermined action intended to minimize the potential system event, wherein the predetermined action taken depends at least on which of the multiple thresholds the frequency of the subset of the system messages exceeded.

2. The method of claim 1, further comprising providing a monitoring and acting system (MAAS), wherein detecting the pattern comprises detecting, by the MAAS, the pattern.

3. The method of claim 1, wherein the predetermined action includes at least one of issuing a notification, initiating a maintenance operation, or shutting down one of the plurality of computing systems.

4. The method of claim 1, further comprising:

identifying, based on the pattern, a statistic to be computed;

querying the message logging database for relevant messages;

computing the statistic based on the relevant messages;

obtaining from the pattern a threshold value and a corresponding action to be taken if the threshold value is surpassed by the computed statistic;

comparing the computed statistic to the threshold value to determine whether the threshold value is surpassed by the computed statistic; and

upon determining that the threshold value is surpassed by the computed statistic, performing the corresponding action.

5. The method of claim 4, further comprising storing the computed statistic in a statistics database.

6. The method of claim 4, wherein the relevant messages consist of messages from a time period.

7. The method of claim 1, wherein the predetermined action comprises shutting down a system from where the system messages originated.

8. A non-transitory computer-readable medium having stored thereon computer-executable instructions causing a processor to effectuate operations, the operations comprising:

storing system messages in a message logging database, the system messages being issued by a plurality of computing systems;

monitoring the system messages stored in the message logging database for predetermined patterns in the system messages;

detecting a pattern in the system messages that indicates a potential system event, wherein the pattern comprises a frequency of a subset of the system messages, wherein the subset of the system messages indicates a potential system event;

determining multiple thresholds corresponding to the frequency of the system messages indicating a potential system event; and

acting in response to the pattern by taking a predetermined action corresponding to the pattern, the predetermined action intended to minimize the potential system event, wherein the predetermined action taken depends at least on which of the multiple thresholds the system message frequency exceeded.

9. The medium of claim 8, wherein detecting the pattern comprises detected, by a monitoring and acting system (MAAS), the pattern.

10. The medium of claim 8, wherein the predetermined action includes at least one of issuing a notification, initiating a maintenance operation, or shutting down one of the plurality of computing systems.

11. The medium of claim 8, the operations further comprising:

identifying, based on the pattern, a statistic to be computed;

querying the message logging database for relevant messages;

computing the statistic based on the relevant messages;

obtaining from the pattern a threshold value and a corresponding action to be taken if the threshold value is surpassed by the computed statistic;

comparing the computed statistic to the threshold value to determine whether the threshold value is surpassed by the computed statistic; and

upon determining that the threshold value is surpassed by the computed statistic, performing the corresponding action.

12. The medium of claim 11, the operations further comprising storing the computed statistic in a statistics database.

13. The medium of claim 11, wherein the relevant messages consist of messages from a time period.

14. The medium of claim 8, wherein the predetermined action comprises shutting down a system from where the system messages originated.

15. A system comprising:

a processor;

memory storing instructions that cause the processor to effectuate operations, the operations comprising;

storing system messages in a message logging database, wherein the system messages are issued by a plurality of computing systems;

monitoring the system messages stored in the message logging database for predetermined patterns in the system messages;

detecting a pattern in the system messages that indicates a potential system event, wherein the pattern comprises a frequency of a subset of the system messages, wherein the subset of the system messages indicates a potential system event;

determining multiple thresholds corresponding to the frequency of the subset of the system messages; and

acting in response to the pattern by taking a predetermined action corresponding to the pattern, the predetermined action intended to minimize the potential system event, wherein the predetermined action taken depends at least on which of the multiple thresholds the frequency of the subset of the system messages exceeded.

16. The system of claim 15, comprising a monitoring and acting system (MAAS).

17. The system of claim 15, wherein the predetermined action includes at least one of issuing a notification, initiating a maintenance operation, or shutting down one of the plurality of computing systems.

18. The system of claim 15, comprising a statistics generator that generates a needed statistic by:

identifying, from the pattern, the statistic to be computed;

querying the message logging database for relevant messages; and

computing the statistic based on the relevant messages,

and comprising a monitoring and acting system (MAAS) that employs the generated statistic by;

obtaining from the pattern a threshold value and a corresponding action to be taken if the threshold value is surpassed by the computed statistic;

comparing the computed statistic to the threshold value to determine whether the threshold value is surpassed by the computed statistic; and

upon determining that the threshold value is surpassed by the computed statistic, performing the corresponding action.

19. The system of claim 18, wherein the statistics generator stores the generated statistic in a statistics database and the MAAS retrieves the generated statistic from the statistics database.

20. The system of claim 15, wherein the predetermined action comprises shutting down the plurality of computing systems.