Log file compression

• US 9,031,997 B2
• Filed: 10/25/2011
• Issued: 05/12/2015
• Est. Priority Date: 10/25/2011
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ramakrishnan, Arun, Shetty, Rohit, Chelliah, Gopikrishnan, Narayanan, Hariharan L.
• Primary Examiner(s): Chbouki, Tarek
• Application Number:
  US13/280,569
• Publication Number:
  US 20130103982A1
• Time in Patent Office: 1,295 Days
• US Class Current: 707/802
• CPC Class Codes: H03M 7/3088 employing the use of a dictionary, e.g. LZ78
• CPC Class Codes: H03M 7/707 Structured documents, e.g. XML

Abstract

A compression system identifies one or more fields in a log file based on at least one field rule from among multiple field rules specified in a log file framework. The compression system extracts contents of the log file associated with the one or more fields. The compression system passes the contents associated with the one or more fields to corresponding compression engines from among a multiple compression engines each specified for performing a separate type of compression from among multiple types of compression for each of the one or more fields, wherein each of the one or more fields corresponds to one or more compression engines from among the multiple compression engines.

Claims

1. A method for compressing a log file, the method comprising:

identifying by one or more processors, two or more fields in a log file based on two or more field rules from among a plurality of field rules specified in a log file framework;

extracting, by the one or more processors, contents of the log file associated with the two or more fields;

passing, by the one or more processors, the contents associated with the two or more fields to two or more corresponding compression engines from among a plurality of compression engines each specified for performing a separate type of compression from among a plurality of types of compression for each of the two or more fields, wherein each of the two or more fields corresponds to one or more compression engines from among the plurality of compression engines;

receiving, by the one or more processors, a plurality of outputs from each of the one or more compression engines, wherein the plurality of outputs comprise compressed contents and uncompressed contents, wherein the compressed contents comprise at least one content element replaced by one or more of the two or more compression engines by a dictionary entry index into at least one entry in a dictionary shared among the two or more compression engines for replacing redundant elements among the two or more compression engines;

identifying, by the one or more processors, the at least one dictionary entry index in the compressed contents, wherein the at least one dictionary entry index is an index value assigned by the one or more compression engines to a text string in the dictionary;

arranging, by the one or more processors, the compressed contents, uncompressed contents, and each text string from the dictionary associated with each identified at least one dictionary entry index in a predefined format for a compressed log file; and

outputting, by the one or more processors, the compressed log file.

2. The method of claim 1, further comprising:

receiving, by the one or more processors, the log file as generated by a log file generator monitoring an application;

identifying, by the one or more processors, the two or more fields in the log file based on the two or more field rules from among the plurality of field rules specified in the log file framework, wherein the plurality of field rules comprise at least one general field rule that specifies rules for one or more general fields common across a plurality of log files created by a plurality of separate applications and at least one application specific field rule that specifies rules for one or more specific fields that are specific to a particular application from among the plurality of separate applications for which the log file is created; and

generating, by the one or more processors, the compressed log file based on the plurality of field rules in the log file framework.

3. The method according to claim 2, further comprising:

accessing, by the one or more processors, from the at least one application specific field rule, at least one resource bundle specifying at least one text string available for formatting messages in the log file, wherein each at least one text string is identified by an index number;

identifying, by the one or more processors, by a particular compression engine specified for indexing static entries from among the plurality of compression engines, the at least one text string within the contents of the log file; and

replacing, by the one or more processors, the at least one text string within the contents of the log file with the index number.

4. The method of claim 1, further comprising compressing, by the one or more processors, the contents associated with each of the two or more fields by applying at least one separate compression rule specified for each type of compression by each of the corresponding compression engines receiving the contents.

5. The method of claim 4, further comprising decompressing, by the one or more processors, the compressed contents of the two or more fields based on predefined decompression rules.

6. The method of claim 1, further comprising compressing, by the one or more processors, the contents associated with two or more fields by each of the plurality of compression engines by replacing the contents with predefined contents.

7. The method of claim 6, wherein replacing, by the one or more processors, the contents further comprises substituting the contents associated with two or more fields with one or more index numbers each assigned to a separate text string.

8. The method according to claim 1, further comprising:

receiving, by a particular compression engine specified for a particular type of compression for at least one particular field from among the two or more fields, contents for the at least one particular field;

applying, by the particular compression engine, at least one particular compression rule specified for the particular compression engine, wherein applying the at least one particular compression rule comprises searching the dictionary for each content element within the contents;

in response to identifying a particular content element within the contents within a particular entry of the dictionary, replacing, by the particular compression engine, the particular content element with an index number of the particular entry in the dictionary;

in response to identify a new content element within the contents that is not included in the dictionary, adding, by the particular compression engine, the new content element as a new entry of the dictionary and replacing the new content element with a new index number of the new entry; and

outputting, by the particular compression engine, compressed content elements of the content comprising the index number and the new index number and one or more uncompressed content elements of the content.

9. A system for compressing a log file, comprising:

one or more processors;

a compression system, for execution by at least one of the one or more processors and operative to identify two or more fields in a log file based on two or more field rules from among a plurality of field rules specified in a log file framework;

the compression system operative to extract contents of the log file associated with the two or more fields;

the compression system operative to pass the contents associated with the two or more fields to two or more corresponding compression engines from among a plurality of compression engines each specified for performing a separate type of compression from among a plurality of types of compression for each of the two or more fields, wherein each of the two or more fields corresponds to one or more compression engines from among the plurality of compression engines;

the compression system operative to receive a plurality of outputs from each of the two or more compression engines, wherein the plurality of outputs comprise compressed contents and uncompressed contents, wherein the compressed contents comprise at least one content element replaced by one or more of the two or more compression engines by a dictionary entry index into at least one entry in a dictionary shared among the two or more compression engines for replacing redundant elements among the two or more compression engines;

the compression system operative to identify the at least one dictionary entry index in the compressed contents, wherein the at least one dictionary entry index is an index value assigned by the one or more compression engines to a text string in the dictionary;

the compression system operative to arrange the compressed contents, uncompressed contents, and each text string from the dictionary associated with each identified at least one dictionary entry index in a predefined format for a compressed log file; and

the compression system operative to output the compressed log file.

10. The system of claim 9, further comprising:

the compression system operative to receive the log file as generated by a log file generator monitoring an application;

the compression system operative to identify the two or more fields in the log file based on the two or more field rules from among the plurality of field rules specified in the log file framework, wherein the plurality of field rules comprise at least one general field rule that specifies rules for one or more general fields common across a plurality of log files created by a plurality of separate applications and at least one application specific field rule that specifies rules for one or more specific fields that are specific to a particular application from among the plurality of separate applications for which the log file is created; and

the compression system operative to generate the compressed log file based on the plurality of field rules in the log file framework.

11. The system of claim 9, further comprising the compression system operative to compress the contents associated with each of the two or more fields by applying at least one separate compression rule specified for each type of compression by each of the corresponding compression engines receiving the contents.

12. The system of claim 11, further comprising the compression system operative to decompress the compressed contents of the two or more fields based on predefined decompression rules.

13. The system of claim 9, further comprising the compression system operative to compress the contents associated with two or more fields by each of the plurality of compression engines by replacing the contents with predefined contents.

14. The system of claim 13, further comprising the compression system operative to replace the contents by substituting the contents associated with two or more fields with one or more index numbers each assigned to a separate text string.

15. A computer program product for compressing a log file, the computer program product comprising:

one or more computer-readable tangible storage devices;

program instructions, stored on at least one of the one or more storage devices, to identify two or more fields in a log file based on two or more field rules from among a plurality of field rules specified in a log file framework;

program instructions, stored on at least one of the one or more storage devices, to extract contents of the log file associated with the two or more fields;

program instructions, stored on at least one of the one or more storage devices, to pass the contents associated with the two or more fields to corresponding compression engines from among a plurality of compression engines each specified for performing a separate type of compression from among a plurality of types of compression for each of the two or more fields, wherein each of the two or more fields corresponds to one or more compression engines from among the plurality of compression engines

program instructions, stored on at least one of the one or more storage devices, to receive a plurality of outputs from each of the two or more compression engines, wherein the plurality of outputs comprise compressed contents and uncompressed contents, wherein the compressed contents comprise at least one content element replaced by one or more of the two or more compression engines by a dictionary entry index into at least one entry in a dictionary shared among the two or more compression engines for replacing redundant elements among the two or more compression engines;

program instructions, stored on at least one of the one or more storage devices, to identify the at least one dictionary entry index in the compressed contents, wherein the at least one dictionary entry index is an index value assigned by the one or more compression engines to a text string in the dictionary;

program instructions, stored on at least one of the one or more storage devices, to arrange the compressed contents, uncompressed contents and each text string from the dictionary associated with each identified at least one dictionary entry index in a predefined format for a compressed log file; and

program instructions, stored on at least one of the one or more storage devices, to output the compressed log file.

16. The computer program product of claim 15, the computer program product further comprising:

program instructions, stored on at least one of the one or more storage devices, to receive the log file as generated by a log file generator monitoring an application;

program instructions, stored on at least one of the one or more storage devices to identify the two or more fields in the log file based on the two or more field rules from among the plurality of field rules specified in the log file framework, wherein the plurality of field rules comprise at least one general field rule that specifies rules for one or more general fields common across a plurality of log files created by a plurality of separate applications and at least one application specific field rule that specifies rules for one or more specific fields that are specific to a particular application from among the plurality of separate applications for which the log file is created; and

program instructions, stored on at least one of the one or more storage devices, to generate the compressed log file based on the plurality of field rules in the log file framework.

17. The computer program product of claim 15, the computer program product further comprising:

program instructions, stored on at least one of the one or more storage devices, to compress the contents associated with each of the two or more fields by applying at least one separate compression rule specified for each type of compression by each of the corresponding compression engines receiving the contents.

18. The computer program product of claim 17, further comprising program instructions, stored on at least one of the one or more storage devices, to decompress the compressed contents of the two or more fields based on predefined decompression rules.

19. The computer program product of claim 17, further comprising program instructions, stored on at least one of the one or more storage devices, to compress the contents associated with two or more fields by each of the plurality of compression engines by replacing the contents with predefined contents.