METHOD, PROGRAM, AND SYSTEM FOR CLASSIFICATION OF SYSTEM LOG

• US 20140324865A1
• Filed: 04/21/2014
• Published: 10/30/2014
• Est. Priority Date: 04/26/2013
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Mizutani, Masayoshi
• Application Number:
  US14/257,100
• Publication Number:
  US 20140324865A1
• Time in Patent Office: Days
• US Class Current: 707/737
• CPC Class Codes: G06F 11/0703 Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36

Abstract

Method and system for classifying system logs. A data processing system reads a message in one line of a system log; prepares a root node of a tree structure in which each node holds a format; calculates a similarity between a log of the root node and the message; generates and stores a first format in the root node if the calculated similarity is equal to or greater than a threshold value; adds the message to a child node of the root node, in accordance with a given condition; searches for, after the first format is created, a second format similar to the first format in a format storage table; combines the first format and the similar format to produce a combined parent format, where the combined parent format holds a plurality of formats; and stores the combined parent format in the format storage table to produce a classified format.

Claims

1. A computer-implemented method for inputting system logs and classifying formats, the method comprising the steps of:

reading a message in one line of a system log;

preparing a root node of a tree structure in which each node holds a format;

calculating a similarity between a log of the root node and the message;

if the calculated similarity is equal to or greater than a threshold value, then

i) generating a first format; and

ii) storing the first format in the root node;

adding the message to a child node of the root node, in accordance with a given condition;

searching for, after the first format is created, a second format that is similar to the first format in a format storage table;

if a similar format is found, then combining the first format and the similar format to produce a combined parent format, wherein the combined parent format holds a plurality of formats; and

storing the combined parent format in the format storage table to produce a classified format.

2. The method according to claim 1, wherein the step of adding the message to a child node of the root node further comprises:

replacing the root node with a most similar child node, if the calculated similarity is less than the threshold value and a number of child nodes held by the root node is equal to or greater than a given number; and

adding the message to the child node of the root node, if the calculated similarity is less than the threshold value and the number of child nodes held by the root node is less than the given number.

3. The method according to claim 1, wherein the step of calculating the similarity between messages further comprises:

dividing the messages into a plurality of sequences to produce divided sequences;

comparing the divided sequences;

adding a score to the divided sequences having a higher similarity; and

dividing a sum of scores by a total number of sequences.

4. The method according to claim 3, wherein if the divided sequences are different, the method includes the step of calculating the similarity between the divided sequences based on a vector of a number of times a character type appears.

5. The method according to claim 1, wherein during the step of searching in the format storage table, an n-gram search is performed.

6. The method according to claim 1, wherein during the combining the first format and the similar format to produce a combined parent format, formats of the plurality are divided into a plurality of editing elements in accordance with a shortest edit script, and each of the plurality of editing elements is processed.

7. A computer readable non-transitory article of manufacture tangibly embodying computer readable instructions, which, when executed, cause a computer to perform the steps of a method for inputting system logs and classifying formats, the method comprising the steps of:

reading a message in one line of a system log;

preparing a root node of a tree structure, wherein each node of the tree structure holds a format;

calculating a similarity between a log of the root node and the message;

if the calculated similarity is equal to or greater than a given threshold, then

i) generating a first format; and

ii) storing the first format in the root node;

adding the message to a child node of the root node, in accordance with a given condition;

searching for, after the first format is created, a second format that is similar to the first format in a format storage table;

if a similar format is found, then combining the first format and the similar format to produce a combined parent format, wherein the combined parent formula holds a combination of a plurality of formats; and

storing the combined parent format in the format storage table to produce a classified format.

8. The article of manufacture according to claim 7, wherein the step of adding the message to a child node of the root node further comprises:

replacing the root node with a most similar child node if the similarity is less than the given threshold and the number of child nodes held by the root node is equal to or greater than a given number; and

adding the message to the child node of the root node, if the similarity is less than the given value and the number of child nodes held by the root node is less than the given number.

9. The article of manufacture according to claim 7, wherein the step of calculating the similarity between messages further comprises:

dividing the messages into a plurality of sequences to produce divided sequences;

comparing at least two of the divided sequences;

adding a score to sequences having a higher similarity; and

dividing a sum of the scores by the number of sequences.

10. The article of manufacture according to claim 9, wherein if different sequences are compared with each other, then calculating the similarity between the sequences on the basis of a vector of a number of times a character type appears.

11. The article of manufacture according to claim 7, wherein during the step of performing searching in the format storage table, an n-gram search is performed.

12. The article of manufacture according to claim 7, wherein during the step of creating the combined parent format, formats are divided into a plurality of editing elements in accordance with a shortest edit script, and each of the plurality of editing elements are processed.

13. A data processing system for inputting system logs and classifying formats, the data processing system comprising a memory and a processing device communicatively coupled to the memory, wherein the processing device is configured to perform the steps of a method comprising:

reading a message in one line of a system log;

preparing a root node of a tree structure, wherein each node of the tree structure holds a format;

calculating a similarity between a log of the root node and the message,

if the calculated similarity is equal to or greater than a given value, then

i) creating a first format; and

ii) storing the first format in the root node;

replacing the root node with a most similar child node if the similarity is less than a given threshold and a number of child nodes held by the root node is equal to or greater than a given number;

adding the message to the child node of the root node, if the similarity is lower than the given threshold and the number of child nodes held by the root node is less than the given number;

searching for, after the new format is created, a second format that is similar to the first format in a format storage table;

if a similar format is found, then combining the new format and the similar format to produce a combined parent format, wherein the combined parent formula holds a combination of a plurality of formats; and

storing the combined parent format in the format storage table to produce a classified format.

14. The data processing system according to claim 13, wherein calculating the similarity between the messages further comprises:

dividing the messages into a plurality of sequences to produce divided sequences;

comparing the divided sequences;

adding a score to sequences having a higher similarity; and

dividing a sum of the scores by a number of sequences.

15. The data processing system according to claim 14, wherein the processing device is further configured to:

calculate a similarity between the sequences using a vector based on a number of times a character type appears, if different sequences are compared with each other.

16. The data processing system according to claim 13, wherein during the searching in the format storage table, an n-gram search is performed.

17. The data processing system according to claim 13, wherein the processing device, during the step of combining the new format and the similar format, is further configured to:

divide formats into a plurality of editing elements in accordance with a shortest edit script; and

process each of the plurality of editing elements.