LOG NORMALIZATION IN ENTERPRISE THREAT DETECTION

• US 20170178026A1
• Filed: 12/22/2015
• Published: 06/22/2017
• Est. Priority Date: 12/22/2015
• Status: Active Application

• Current Assignee: SAP SE
• Original Assignee: SAP SE
• Inventors: Merkel, Rita, Rodeck, Marco, Carullo, Lukas, Thomas, Susan Marie, Bersch, Viktor, Kunz, Thomas, Chrosziel, Florian, Al-Hujaj, Omar Alexander, Mehta, Harish, Seifert, Hartwig
• Application Number:
  US14/978,995
• Publication Number:
  US 20170178026A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06N 5/046 Forward inferencing; Production systems

Abstract

A sample log file including a plurality of log entries for log learning is accessed, using a log interpretation controller, prior to runtime as part of a log learning process. Each of the plurality of log entries is analyzed. A log entry type is assigned to each of the plurality of log entries. A log type and semantic event are assigned to each log entry type. Generation of runtime rules is triggered for analyzing unknown log entries. The runtime rules include characteristics of particular log entry types that allow unique identification of the particular log entry type for a particular unknown log entry. The generated runtime rules are loaded into a runtime parser.

Claims

1. A computer-implemented method, comprising:

prior to runtime as part of a log learning process;

accessing, using a log interpretation controller, a sample log file including a plurality of log entries for log learning;

analyzing each of the plurality of log entries;

assigning a log entry type to each of the plurality of log entries;

assigning a log type and semantic event to each log entry type;

triggering generation of runtime rules for analyzing unknown log entries wherein the runtime rules include characteristics of particular log entry types that allow unique identification of the particular log entry type for a particular unknown log entry; and

loading the generated runtime rules into a runtime parser.

2. The method of claim 1, comprising identifying components of each log entry, the components including one or more of internet protocol (IP) address, timestamp, hostname, and media access control (MAC) address.

3. The method of claim 1, wherein each log entry type is a group of one or more log entries with identical internal structure.

4. The method of claim 1, comprising assigning attributes of a knowledge base to each log entry type.

5. The method of claim 4, wherein the runtime rules also include regular expressions to extract parts of the particular unknown log entry and to assign the extracted parts to attributes of the knowledge base.

6. The method of claim 1, comprising, at runtime:

parsing runtime log entries from an external log source with a runtime parser to identify recognized runtime log entries and unrecognized runtime log entries;

storing the unrecognized runtime log entries; and

for each recognized runtime log entry;

assigning the runtime log entry with a log entry type;

assigning the runtime log entry with a semantic event type; and

extracting the runtime log entry into a threat detection tool for forensic analysis.

7. The method of claim 6, comprising re-processing the stored unrecognized runtime log entries to generate updated runtime rules to correctly recognize the unrecognized runtime log entries.

8. A non-transitory, computer-readable medium storing computer-readable instructions, the instructions executable by a computer and configured to, prior to runtime as part of a log learning process:

access, using a log interpretation controller, a sample log file including a plurality of log entries for log learning;

analyze each of the plurality of log entries;

assign a log entry type to each of the plurality of log entries;

assign a log type and semantic event to each log entry type;

trigger generation of runtime rules for analyzing unknown log entries wherein the runtime rules include characteristics of particular log entry types that allow unique identification of the particular log entry type for a particular unknown log entry; and

load the generated runtime rules into a runtime parser.

9. The non-transitory, computer-readable medium of claim 8, the instructions further configured to identify components of each log entry, the components including one or more of internet protocol (IP) address, timestamp, hostname, and media access control (MAC) address.

10. The non-transitory, computer-readable medium of claim 8, wherein each log entry type is a group of one or more log entries with identical internal structure.

11. The non-transitory, computer-readable medium of claim 8, the instructions further configured to assign attributes of a knowledge base to each log entry type.

12. The non-transitory, computer-readable medium of claim 11, wherein the runtime rules also include regular expressions to extract parts of the particular unknown log entry and to assign the extracted parts to attributes of the knowledge base.

13. The non-transitory, computer-readable medium of claim 8, the instructions further configured to, at runtime:

parse runtime log entries from an external log source with a runtime parser to identify recognized runtime log entries and unrecognized runtime log entries;

store the unrecognized runtime log entries; and

for each recognized runtime log entry;

assign the runtime log entry with a log entry type;

assign the runtime log entry with a semantic event type; and

extract the runtime log entry into a threat detection tool for forensic analysis.

14. The non-transitory, computer-readable medium of claim 13, the instructions further configured to re-process the stored unrecognized runtime log entries to generate updated runtime rules to correctly recognize the unrecognized runtime log entries.

15. A system, comprising:

a memory;

at least one hardware processor interoperably coupled with the memory and configured to, prior to runtime as part of a log learning process;

access, using a log interpretation controller, a sample log file including a plurality of log entries for log learning;

analyze each of the plurality of log entries;

assign a log entry type to each of the plurality of log entries;

assign a log type and semantic event to each log entry type;

trigger generation of runtime rules for analyzing unknown log entries wherein the runtime rules include characteristics of particular log entry types that allow unique identification of the particular log entry type for a particular unknown log entry; and

load the generated runtime rules into a runtime parser.

16. The system of claim 15, the processor further configured to identify components of each log entry, the components including one or more of internet protocol (IP) address, timestamp, hostname, and media access control (MAC) address.

17. The system of claim 15, wherein each log entry type is a group of one or more log entries with identical internal structure.

18. The system of claim 15, the processor further configured to assign attributes of a knowledge base to each log entry type.

19. The system of claim 18, wherein the runtime rules also include regular expressions to extract parts of the particular unknown log entry and to assign the extracted parts to attributes of the knowledge base.

20. The system of claim 15, the processor further configured to, at runtime:

parse runtime log entries from an external log source with a runtime parser to identify recognized runtime log entries and unrecognized runtime log entries;

store the unrecognized runtime log entries; and

for each recognized runtime log entry;

assign the runtime log entry with a log entry type;

assign the runtime log entry with a semantic event type; and

extract the runtime log entry into a threat detection tool for forensic analysis.