LOG FILE COMPRESSION

• US 20130103982A1
• Filed: 10/25/2011
• Published: 04/25/2013
• Est. Priority Date: 10/25/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: SHETTY, ROHIT, CHELLIAH, GOPIKRISHNAN, NARAYANAN, HARIHARAN L., RAMAKRISHNAN, ARUN
• Granted Patent: US 9,031,997 B2
• Time in Patent Office: Days
• US Class Current: 714/20
• CPC Class Codes: H03M 7/3088 employing the use of a dictionary, e.g. LZ78
• CPC Class Codes: H03M 7/707 Structured documents, e.g. XML

Abstract

A compression system identifies one or more fields in a log file based on at least one field rule from among multiple field rules specified in a log file framework. The compression system extracts contents of the log file associated with the one or more fields. The compression system passes the contents associated with the one or more fields to corresponding compression engines from among a multiple compression engines each specified for performing a separate type of compression from among multiple types of compression for each of the one or more fields, wherein each of the one or more fields corresponds to one or more compression engines from among the multiple compression engines.

Claims

1. A method for compressing a log file, the method comprising:

identifying one or more fields in a log file based on at least one field rule from among a plurality of field rules specified in a log file framework;

extracting contents of the log file associated with the one or more fields; and

passing the contents associated with the one or more fields to corresponding compression engines from among a plurality of compression engines each specified for performing a separate type of compression from among a plurality of types of compression for each of the one or more fields, wherein each of the one or more fields corresponds to one or more compression engines from among the plurality of compression engines.

2. The method of claim 1, further comprising:

receiving the log file as generated by a log file generator monitoring an application and generating the log file based on the plurality of field rules in the log file framework.

3. The method of claim 1, further comprising compressing the contents associated with each of the one or more fields by applying at least one separate compression rule specified for each type of compression by each of the corresponding compression engines receiving the contents.

4. The method of claim 3, further comprising:

receiving, by a log formatter, a plurality of outputs from each of the one or more compression engines, wherein the plurality of outputs comprise compressed contents and uncompressed contents;

identifying at least one dictionary entry index in the compressed contents, wherein the at least one dictionary entry index is an index value assigned by the one or more compression engines to a text string in a dictionary; and

arranging the compressed contents, uncompressed contents, and identified at least one dictionary entry index in a predefined format for a compressed log file.

5. The method of claim 3, further comprising decompressing the compressed contents of the one or more fields based on predefined decompression rules.

6. The method of claim 1, further comprising compressing the contents associated with one or more fields by each of the plurality of compression engines by replacing the contents with predefined contents.

7. The method of claim 6, wherein replacing the contents further comprises substituting the contents associated with one or more fields with one or more index numbers each assigned to a separate text string.

8. A system for compressing a log file, comprising:

one or more processors;

a compression system, for execution by at least one of the one or more processors and operative to identify one or more fields in a log file based on at least one field rule from among a plurality of field rules specified in a log file framework;

the compression system operative to extract contents of the log file associated with the one or more fields; and

the compression system operative to pass the contents associated with the one or more fields to corresponding compression engines from among a plurality of compression engines each specified for performing a separate type of compression from among a plurality of types of compression for each of the one or more fields, wherein each of the one or more fields corresponds to one or more compression engines from among the plurality of compression engines.

9. The system of claim 8, further comprising:

the compression system operative to receive the log file as generated by a log file generator monitoring an application and generating the log file based on the plurality of field rules in the log file framework.

10. The system of claim 8, further comprising the compression system operative to compress the contents associated with each of the one or more fields by applying at least one separate compression rule specified for each type of compression by each of the corresponding compression engines receiving the contents.

11. The system of claim 10, further comprising:

the compression system operative to receive a plurality of outputs from each of the one or more compression engines, wherein the plurality of outputs comprise compressed contents and uncompressed contents;

the compression system operative to identify at least one dictionary entry index in the compressed contents, wherein the at least one dictionary entry index is an index value assigned by the one or more compression engines to a text string in a dictionary; and

the compression system operative to arrange the compressed contents, uncompressed contents, and identified at least one dictionary entry index in a predefined format for a compressed log file.

12. The system of claim 10, further comprising the compression system operative to decompress the compressed contents of the one or more fields based on predefined decompression rules.

13. The system of claim 8, further comprising the compression system operative to compress the contents associated with one or more fields by each of the plurality of compression engines by replacing the contents with predefined contents.

14. The system of claim 13, further comprising the compression system operative to replace the contents by substituting the contents associated with one or more fields with one or more index numbers each assigned to a separate text string.

15. A computer program product for compressing a log file, the computer program product comprising:

one or more computer-readable tangible storage devices;

program instructions, stored on at least one of the one or more storage devices, to identify one or more fields in a log file based on at least one field rule from among a plurality of field rules specified in a log file framework;

program instructions, stored on at least one of the one or more storage devices, to extract contents of the log file associated with the one or more fields; and

program instructions, stored on at least one of the one or more storage devices, to pass the contents associated with the one or more fields to corresponding compression engines from among a plurality of compression engines each specified for performing a separate type of compression from among a plurality of types of compression for each of the one or more fields, wherein each of the one or more fields corresponds to one or more compression engines from among the plurality of compression engines.

16. The computer program product of claim 15, the computer program product further comprising:

program instructions, stored on at least one of the one or more storage devices, to receive the log file as generated by a log file generator monitoring an application and generating the log file based on the plurality of field rules in the log file framework.

17. The computer program product of claim 15, the computer program product further comprising:

program instructions, stored on at least one of the one or more storage devices, to compress the contents associated with each of the one or more fields by applying at least one separate compression rule specified for each type of compression by each of the corresponding compression engines receiving the contents.

18. The computer program product of claim 17, the computer program product further comprising:

program instructions, stored on at least one of the one or more storage devices, to receive a plurality of outputs from each of the one or more compression engines, wherein the plurality of outputs comprise compressed contents and uncompressed contents;

program instructions, stored on at least one of the one or more storage devices, to identify at least one dictionary entry index in the compressed contents, wherein the at least one dictionary entry index is an index value assigned by the one or more compression engines to a text string in a dictionary; and

program instructions, stored on at least one of the one or more storage devices, to arrange the compressed contents, uncompressed contents, and identified at least one dictionary entry index in a predefined format for a compressed log file.

19. The computer program product of claim 17, further comprising program instructions, stored on at least one of the one or more storage devices, to decompress the compressed contents of the one or more fields based on predefined decompression rules.

20. The computer program product of claim 17, further comprising program instructions, stored on at least one of the one or more storage devices, to compress the contents associated with one or more fields by each of the plurality of compression engines by replacing the contents with predefined contents.