Method and apparatus for compressing log record information

• US 20050138483A1
• Filed: 03/26/2002
• Published: 06/23/2005
• Est. Priority Date: 03/26/2002
• Status: Active Grant

• Current Assignee: RPX Corporation
• Original Assignee: Nokia Solutions and Networks US LLC (Nokia Corporation)
• Inventors: Hatonen, Kimmo, Miettinen, Markus
• Granted Patent: US 7,778,979 B2
• Time in Patent Office: Days
• US Class Current: 714/45
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 2221/2101 Auditing as a secondary aspect

Abstract

The present invention relates to a method and apparatus for compressing a log record information provided e.g. to a monitoring system. Frequent patterns in the log information are detected and, then, redundant frequent patterns whose value or record combination is a subset of a value or record combination of another one of said detected frequent patterns are removed. Accordingly, a general method is provided which can be applied to all event logs arising from communication net-works or other monitoring systems. The detection of frequent patterns is only based on value or record combinations and is thus independent of the specific application. On the other hand, the size of the stored log files can be decreased remarkably while remaining readable for human beings.

Claims

1. A method of compressing a log record information, said method comprising the steps of:

a) detecting frequent patterns in said log information; and

b) removing redundant frequent patterns whose value or record combination is a subset of a value or record combination of another one of said detected frequent patterns.

2. A method according to claim 1, wherein said frequent patterns correspond to frequently occurring values and/or frequently occurring records of said log record information.

3. A method according to claim 2, wherein said values comprise constant field, set or sequence values of a set of said log record information.

4. A method according to claim 2, wherein said records comprise frequent line sequences of said log record information.

5. A method according to claim 1, wherein said detection step comprises counting occurrences of patterns, and comparing the counting result with a predetermined threshold.

6. A method according to claim 1, wherein said removing step is used to create a condensed representation of said log record information.

7. A method according to claim 6, wherein said creation of said condensed representation comprises the steps of:

ordering said detected frequent patterns to a linear order where a superset precedes its subsets until no more detected frequent patterns are left; and

removing all frequent patterns that are subsets of the first pattern and which occur in said log record information when said first pattern does.

8. A method according to claim 7, wherein said removing step of said creation comprises the step of decreasing a support information allocated to a subset of said first pattern, if said subset also occurs in said log record information when said first pattern does not, by a support information allocated to said first pattern, and removing said subset if its support information becomes zero.

9. A method according to claim 6, further comprising the step of creating an adequate information of said log record information by removing or marking lines or line combinations which support said detected frequent patterns, and adding said condensed representation.

10. A method according to claim 6, further comprising the step of creating an adequate information log by linking said detected and compressed frequent patterns to said log record information.

11. A method according to claim 6, further comprising the step of creating an adequate information by constructing an inclusion hierarchy of said detected and compressed frequent patterns, and ordering said detected and compressed frequent patterns using an estimation function.

12. A method according to claim 6, further comprising the step of creating an adequate information by extracting association or episode rules and linking an inclusion hierarchy of said detected and compressed frequent patterns to said extracted rules.

13. A method according to claim 1, further comprising the step of restricting said detection step by setting constraints on predetermined field or field combinations of said log record information.

14. A method according to claim 1, wherein said removing step is used to create a condensed directory structure for separating and coding said log record information in order to be stored in a log file.

15. A method according to claim 14, further comprising the step of applying a query first to said detected and compressed frequent patterns and then to unmatched portions of said log file.

16. A method according to claim 15, wherein all lines of a pattern are returned if said pattern matches with said query.

17. A method according to claim 14, further comprising the step of grouping reduced lines of said log file by said detected and compressed frequent patterns.

18. A method according to claim 14, further comprising the step of adding a pattern identification to lines of said log file, from which fields already included in said detected and compressed frequent patterns have been removed.

19. A method according to claim 14, further comprising the step of attaching a regular expression or a query to each of said detected and compressed frequent patterns, said regular expression matching only to those lines of said log file, where the values included in the corresponding pattern are present.

20. A method according to claim 19, wherein said removing step comprises the steps of:

reading a file header of said log file;

removing constant values from each line;

if said regular expression of a detected and compressed frequent pattern matches to a line, removing all values included in said pattern from said line and adding said line to a list of lines matched by said pattern; and

adding remaining lines of said log file to a list of unmatched lines.

21. A method according to claim 19, wherein said lists of matched lines and unmatched lines are appended to individual printouts of lines.

22. A method according to claim 14, further comprising the step of adding markup language tags to said log file.

23. A method according to claim 1, wherein said log record information is provided to a monitoring system.

28. A computer program product adapted to perform the steps of claim 1 when being loaded to a data processing system.

24. An apparatus for compressing a log record information, said apparatus comprising:

a) detecting means for detecting frequent patterns in said log information,

b) compressing means for removing redundant frequent patterns whose value or record combination is a subset of a value or record combination of another are of said detected frequent patterns.

25. An apparatus according to claim 24, wherein said apparatus is arranged to create a condensed representation of said log record information.

26. An apparatus according to claim 24, wherein said apparatus is arranged to create a condensed directory structure for separating and coding said log record information in order to be stored in a log file.

27. An apparatus according to claim 24, wherein said apparatus is arranged to provide said log record information to a monitoring system.