Compression of Log Data Using Pattern Recognition

• US 20200134029A1
• Filed: 10/29/2018
• Published: 04/30/2020
• Est. Priority Date: 10/29/2018
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Savir, Amihai, Natanzon, Assaf, Gefen, Avitan
• Granted Patent: US 11,151,089 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/174 Redundancy elimination performed by the file system management of the data involved in backup or backup restore using de-duplication of the data G06F11/14
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

Techniques are provided for compression and decompression of log data using pattern recognition. An exemplary method comprises: obtaining at least one log message, wherein the at least one log message comprises a pattern-based message template and one or more message variables, wherein the pattern-based message template was learned from historical log data using a log pattern mining technique; obtaining a compression index that maps a plurality of pattern-based message templates to a corresponding message signature; and writing the one or more message variables and a message signature corresponding to the pattern-based message template of the at least one log message to a log file. The log pattern mining technique learns a given pattern-based message template from the historical log data to represent a plurality of historical log messages. The at least one log message may comprise a historical log message or may be from a real-time stream of log messages.

Claims

1. A method, comprising:

obtaining at least one log message, wherein the at least one log message comprises a pattern-based message template and one or more message variables, wherein the pattern-based message template was learned from historical log data using a log pattern mining technique;

obtaining a compression index that maps a plurality of pattern-based message templates to a corresponding message signature; and

writing, using at least one processing device, the one or more message variables and a message signature corresponding to the pattern-based message template of the at least one log message to a log file.

2. The method of claim 1, wherein the log pattern mining technique learns a given pattern-based message template from the historical log data to represent a plurality of historical log messages.

3. The method of claim 1, further comprising the step of incrementing a counter associated with each of a plurality of distinct pattern-based message templates, wherein a given message signature is assigned to a particular pattern-based message template based on a length of the given message signature and a frequency of occurrence of the particular pattern-based message template.

4. The method of claim 1, further comprising the steps of determining whether the pattern-based message template is in the compression index and performing the following steps when a given pattern-based message template is not in the compression index:

adding a message signature to the compression index and a decompression index for the given pattern-based message template; and

initializing the counter for the given pattern-based message template.

5. The method of claim 1, wherein the compression index comprises a key/value database where a plurality of the pattern-based message templates are keys and the corresponding message signatures are values of the key/value database.

6. The method of claim 1, further comprising the step of decompressing the log file using a decompression index that maps a plurality of message signatures to corresponding pattern-based message templates.

7. The method of claim 6, wherein the decompression index comprises a key/value database where the message signatures are keys and the corresponding plurality of pattern-based message templates are values of the key/value database.

8. The method of claim 1, wherein the at least one log message is from one or more historical log messages and a real-time stream of log messages.

9. A system, comprising:

a memory; and

at least one processing device, coupled to the memory, operative to implement the following steps;

obtaining at least one log message, wherein the at least one log message comprises a pattern-based message template and one or more message variables, wherein the pattern-based message template was learned from historical log data using a log pattern mining technique;

obtaining a compression index that maps a plurality of pattern-based message templates to a corresponding message signature; and

writing the one or more message variables and a message signature corresponding to the pattern-based message template of the at least one log message to a log file.

10. The system of claim 9, wherein the log pattern mining technique learns a given pattern-based message template from the historical log data to represent a plurality of historical log messages.

11. The system of claim 9, further comprising the step of incrementing a counter associated with each of a plurality of distinct pattern-based message templates, wherein a given message signature is assigned to a particular pattern-based message template based on a length of the given message signature and a frequency of occurrence of the particular pattern-based message template.

12. The system of claim 9, further comprising the steps of determining whether the pattern-based message template is in the compression index and performing the following steps when a given pattern-based message template is not in the compression index:

adding a message signature to the compression index and a decompression index for the given pattern-based message template; and

initializing the counter for the given pattern-based message template.

13. The system of claim 9, further comprising the step of decompressing the log file using a decompression index that maps a plurality of message signatures to corresponding pattern-based message templates.

14. The system of claim 9, wherein the at least one log message is from one or more historical log messages and a real-time stream of log messages.

15. A computer program product, comprising a tangible machine-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by at least one processing device perform the following steps:

obtaining at least one log message, wherein the at least one log message comprises a pattern-based message template and one or more message variables, wherein the pattern-based message template was learned from historical log data using a log pattern mining technique;

obtaining a compression index that maps a plurality of pattern-based message templates to a corresponding message signature; and

writing the one or more message variables and a message signature corresponding to the pattern-based message template of the at least one log message to a log file.

16. The computer program product of claim 15, wherein the log pattern mining technique learns a given pattern-based message template from the historical log data to represent a plurality of historical log messages.

17. The computer program product of claim 15, further comprising the step of incrementing a counter associated with each of a plurality of distinct pattern-based message templates, wherein a given message signature is assigned to a particular pattern-based message template based on a length of the given message signature and a frequency of occurrence of the particular pattern-based message template.

18. The computer program product of claim 15, further comprising the steps of determining whether the pattern-based message template is in the compression index and performing the following steps when a given pattern-based message template is not in the compression index:

adding a message signature to the compression index and a decompression index for the given pattern-based message template; and

initializing the counter for the given pattern-based message template.

19. The computer program product of claim 15, further comprising the step of decompressing the log file using a decompression index that maps a plurality of message signatures to corresponding pattern-based message templates.

20. The computer program product of claim 15, wherein the at least one log message is from one or more historical log messages and a real-time stream of log messages.