Systems and methods for automated log event normalization using three-staged regular expressions

• US 8,079,081 B1
• Filed: 06/27/2008
• Issued: 12/13/2011
• Est. Priority Date: 06/27/2008
• Status: Active Grant

• Current Assignee: Alert Logic Incorporated
• Original Assignee: Alert Logic Incorporated
• Inventors: Golovinsky, Eugene, Lavrik, Anton, Trakhtman, Pavel, Fisher, Paul
• Primary Examiner(s): Barron, Jr., Gilberto
• Assistant Examiner(s): Cribbs, Malcolm
• Application Number:
  US12/163,733
• Time in Patent Office: 1,264 Days
• Field of Search: 726/12, 726/13, 726/22, 713/168, 713/176, 713/178
• US Class Current: 726/22
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 63/102 Entity profiles
• CPC Class Codes: H04L 63/12 Applying verification of the received information cryptographic mechanisms or cryptographic arrangements for data integrity or data verification H04L9/32
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection

Abstract

Methods and systems for normalizing log messages. Some methods include obtaining a freeform log message from one of many disparate programs. The methods can include determining which program originated the message and, based on that, determining a signature which matches the message. Using the signature, a parsing expression may be determined with which to extract information from a portion of the message. The time from obtaining the message to extracting the information can be about the same for all messages and can be about 1/40,000^th of a second. In some embodiments, a generic signature of the message may be output. A version of the message may be reconstructed based on the generic signature and information. When more than one message signatures matches the reconstructed message, one of the matching signatures can be adjusted. The parsing expression can be the first of an ordered list of expressions which successfully evaluates the log message.

Claims

1. A method of processing log messages, the method comprising:

at a data center computer;

at a first stage, partially parsing a freeform log message to identify a program group from which the freeform log message is originated;

at a second stage, partially parsing the freeform log message to identify at type of the freeform log message which is associated with a message signature which, in turn, is associated with the program group from which the freeform log message is originated;

determining a parsing expression based on the message signature;

at a third stage, partially or completely parsing at least a portion of the freeform log message using the parsing expression to extract information from the freeform log message; and

outputting the extracted information.

2. The method of claim 1 wherein a time to extract the information is about the same for any of the freeform log messages.

3. The method of claim 2 wherein the time is about 1/40,000 of a second.

4. The method of claim 1 further comprising outputting a generic signature of the freeform log message.

5. The method of claim 4 further comprising:

reconstructing a version of the freeform log message based on the generic signature and the extracted information;

comparing the reconstructed version of the log message to a plurality of message signatures; and

when more than one message signature matches the reconstructed version of the log message, adjusting at least one of the matching message signatures.

6. The method of claim 1 wherein the parsing expression is one of an ordered list of parsing expressions associated with the message signature and the determining the parsing expression further comprises evaluating each of the parsing expressions in the order of the ordered list and using the first parsing expression that successfully evaluates for the parsing a portion of the freeform log message.

7. The method of claim 1 wherein the extracted information includes one or more token type, value pairs.

8. A non-transitory computer readable medium carrying instructions for processing log messages which when executed by a machine cause the machine to:

partially parse a freeform log message to identify a program group from which the freeform log message is originated;

partially parse the freeform log message to identify a type of the freeform log message which is associated with a message signature which, in turn, is associated with the program group from which the freeform log message is originated;

determine a parsing expression based on the message signature;

partially or completely parse at least a portion of the freeform log message using the parsing expression to extract information from the freeform log message; and

output the extracted information.

9. The non-transitory computer readable medium of claim 8, wherein a time to extract the information is about the same for any of the freeform log messages.

10. The non-transitory computer readable medium of claim 9, wherein the time is about 1/40,000 of a second.

11. The non-transitory computer readable medium of claim 8, wherein the instructions further comprise instructions which when executed cause the machine to output a generic signature of the freeform log message.

12. The non-transitory computer readable medium of claim 11, wherein the instructions further comprise instructions which when executed cause the machine to:

reconstruct a version of the freeform log message based on the generic signature and the extracted information;

compare the reconstructed version of the log message to a plurality of message signatures; and

when more than one message signature matches the reconstructed version of the log message, adjust at least one of the matching message signatures.

13. The non-transitory computer readable medium of claim 8, wherein the parsing expression is one of an ordered list of parsing expressions associated with the message signature and the instructions which cause the machine to determine the parsing expression further comprise instructions which cause the machine to evaluate each of the parsing expressions in the order of the ordered list and use the first parsing expression that successfully evaluates for parsing a portion of the freeform log message.

14. The non-transitory computer readable medium of claim 8, wherein the extracted information includes one or more token type, value pairs.

15. A system for processing log messages comprising:

a processor;

an interface; and

a machine readable medium, the processor, the interface, and the machine readable medium being in communication with each other, the machine readable medium carrying instructions for processing log messages which when executed by the processor cause the processor to;

partially parse a freeform log message to identify a program group from which the freeform log message is originated;

partially parse the freeform log message to identify a type of the freeform loq message which is associated with a message signature which, in turn, is associated with the program group from which the freeform log message is originated;

determine a parsing expression based on the message signature;

partially or completely parse at least a portion of the freeform log message using the parsing expression to extract information from the freeform log message; and

output the extracted information.

16. The system of claim 15 wherein a time to extract the information is about the same for any of the freeform log messages.

17. The system of claim 16 wherein the time is about 1/40,000 of a second.

18. The system of claim 15 wherein the instructions further comprise instructions which when executed cause the processor to output a generic signature of the freeform log message.

19. The system of claim 18 wherein the instructions further comprise instructions which when executed cause the processor to:

reconstruct a version of the freeform log message based on the generic signature and the extracted information;

compare the reconstructed version of the log message to a plurality of message signatures; and

when more than one message signature matches the reconstructed version of the log message, adjust at least one of the matching message signatures.

20. The system of claim 19 wherein the parsing expression is one of an ordered list of parsing expressions associated with the message signature and the instructions which cause the processor to determine the parsing expression further comprise instructions which cause the machine to evaluate each of the parsing expressions in the order of the ordered list and use the first parsing expression that successfully evaluates for parsing a portion of the freeform log message.