Apparatus for and Method of Implementing system Log Message Ranking via System Behavior Analysis

• US 20090113246A1
• Filed: 10/24/2007
• Published: 04/30/2009
• Est. Priority Date: 10/24/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Tsherniak, Aviad, Yom-Tov, Elad, Sabato, Sivan
• Granted Patent: US 8,452,761 B2
• Time in Patent Office: Days
• US Class Current: 714/37
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/0769 Readable error formats, e.g. cross-platform generic formats, human understandable formats
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
• CPC Class Codes: G06F 11/328 Computer systems status display G06F11/327 takes precedence
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 2201/88 Monitoring involving counting

Abstract

A novel and useful method for enabling system logs to be effectively and efficiently monitored by ranking the system log messages by their estimated value to administrators and generating a log view that displays the most important messages. The ranking process uses a dataset of system logs from many computer systems to score messages. For better scoring, unsupervised clustering is used to identify sets of systems that behave similarly. The expected distribution of messages in a given system is estimated using the resulting clusters, and log messages are scored using this estimation.

Claims

1. A method of analyzing system logs, said method comprising the steps of:

creating at least one system profile representing a type of system;

matching a system log to be analyzed to the most similar system profile;

calculating a score for each system log message from said system log to be analyzed; and

ranking said scored plurality of system log message in order to identify any atypical system log messages.

2. The method according to claim 1, wherein said analysis is performed at a centralized help desk responsible for supporting one or more systems.

3. The method according to claim 1, wherein said system log analysis is performed at a remote system to be analyzed.

4. The method according to claim 1, wherein said step of creating at least one system profile comprises the steps of:

collecting a plurality of system logs;

preprocessing messages from said system logs into a canonical form;

creating a count vector for each system log representing the frequency that each said preprocessed message appears in said system log;

clustering said count vectors into said one or more system profiles; and

calculating an average count vector for said one or more system profiles representing the average frequency that each said preprocessed appears in said count vectors of said profile.

5. The method according to claim 2, wherein said step of clustering comprises using a correlation coefficient to measure similarity between each pair of said count vectors.

6. The method according to claim 1, wherein said step of matching a system log to be analyzed to the most similar system profile comprises the steps of:

preprocessing said system log messages into a canonical form;

creating a count vector for said system log representing the frequency that each said preprocessed message appears in said system log;

identifying the system profile most similar to said count vector; and

scoring said count vector.

7. The method according to claim 6, wherein said step of identifying comprises using a correlation coefficient to measure similarity between said count vector of said system log to be analyzed and said average count vector of said system profile.

8. The method according to claim 1, wherein said score represents a relationship between said number representing the observed frequency of said preprocessed message from said system log to be analyzed and said number representing the observed frequency of said preprocessed message from said system profile.

9. A method of defining one or more system profiles for use in the analysis of system logs, said method comprising the steps of:

collecting a plurality of system logs;

preprocessing messages from said system log into a canonical form;

creating a count vector for each system log representing the frequency that each said preprocessed message appears in said system log;

clustering said count vectors into said one or more system profiles; and

calculating an average count vector for said one or more system profiles representing the average frequency that each said preprocessed appears in said count vectors of said profile.

10. The method according to claim 9, wherein said step of clustering comprises using a correlation coefficient to measure similarity between each pair of said count vectors.

11. A method of ranking system log messages according to their severity, said method comprising the steps of:

preprocessing said system log messages into a canonical form;

creating a count vector from said preprocessed system log messages representing the frequency that each said preprocessed message appears;

matching said count vector to a system profile; and

calculating a score for each preprocessed system log message.

12. The method according to claim 11, wherein said system profile comprises an average count vector representing the expected frequency of each said preprocessed message.

13. The method according to claim 11, wherein said step of matching comprises using a correlation coefficient to measure similarity between said count vector and said system profile.

14. The method according to claim 11, wherein said score represents a relationship between said number representing the observed frequency of said preprocessed messages and said number representing the observed frequency of said preprocessed messages from said system profile.

15. A computer program comprising:

a computer usable medium having computer usable program code for analyzing system log messages;

said computer program product including;

computer usable program code for creating at least one system profile representing a type of system;

computer usable program code for matching a system log to be analyzed to the most similar system profile;

computer usable program code for calculating a score for each system log message from said system log to be analyzed; and

computer usable program code for ranking said scored system log messages to identify any atypical system log messages.

16. The method according to claim 15, wherein said step of creating at least one system profile comprises the steps of:

collecting a plurality of system logs;

preprocessing messages from said system logs into a canonical form;

creating a count vector for each system log representing the frequency that each said preprocessed message appears in said system log;

clustering said count vectors into said one or more system profiles; and

calculating an average count vector for said one or more system profiles representing the average frequency that each preprocessed appears in said count vectors of said profile.

17. The method according to claim 15, wherein said step of clustering comprises using a correlation coefficient to measure similarity between each pair of said count vectors.

18. The method according to claim 15, wherein said step of matching a system log to be analyzed to the most similar system profile comprises the steps of:

preprocessing said system log messages into a canonical form;

creating a count vector for said system log representing the frequency that each said preprocessed message appears in said system log;

identifying the system profile most similar to said count vector; and

scoring said count vector.

19. The method according to claim 15, wherein said step of identifying comprises using a correlation coefficient to measure similarity between said count vector of said system log to be analyzed and said average count vector of said system profile.

20. The method according to claim 15, wherein said score represents a relationship between said number representing the observed frequency of said preprocessed message from said system log to be analyzed and said number representing the observed frequency of said preprocessed message from said system profile.