Automated event ID field analysis on heterogeneous logs

• US 10,237,295 B2
• Filed: 02/10/2017
• Issued: 03/19/2019
• Est. Priority Date: 03/22/2016
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Zhang, Hui, Jiang, Guofei
• Primary Examiner(s): Paliwal, Yogesh
• Application Number:
  US15/429,849
• Publication Number:
  US 20170279840A1
• Time in Patent Office: 767 Days
• Field of Search: 726 23
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

A system, program, and method for anomaly detection in heterogeneous logs. The system having a processor configured to identify pattern fields comprised of a plurality of event identifiers. The processor is further configured to generate an automata model by profiling event behaviors of the plurality of event sequences, the plurality of event sequences grouped in the automata model by combinations of one or more pattern fields and one or more event identifiers from among the plurality of event identifiers, wherein for a given combination, the one or more event identifiers therein must be respectively comprised in a same one of the one or more pattern fields with which it is combined. The processor is additionally configured to detect an anomaly in one of the plurality of event sequences using the automata model. The processor is also configured to control an anomaly-initiating one of the network devices based on the anomaly.

Claims

1. A method performed in a network having network devices, including computers, that generate heterogeneous logs which include a plurality of event sequences, the method comprising:

identifying, by a processor from the heterogeneous logs, pattern fields comprised of a plurality of event identifiers;

generating, by the processor, an automata model by profiling event behaviors of the plurality of event sequences, the plurality of event sequences grouped in the automata model by combinations of one or more pattern fields and one or more event identifiers from among the plurality of event identifiers, wherein for a given combination, the one or more event identifiers therein must be respectively comprised in a same one of the one or more pattern fields with which it is combined;

detecting, by the processor, an anomaly in one of the plurality of event sequences using the automata model; and

controlling, by the processor, an anomaly-initiating one of the network devices based on the anomaly

wherein the identifying pattern fields comprises;

performing a tokenization process on the heterogeneous logs to generate tokens;

performing a log similarity process on the heterogeneous logs based on the tokens to identify log similarities amongst the heterogeneous logs; and

clustering the heterogeneous logs based on the log similarities to obtain clustered heterogeneous logs; and

wherein the identifying pattern fields further comprises;

aligning the clustered heterogeneous logs to preserve unknown layouts;

discovering a log motif to find a most representative layout from the unknown layouts and a plurality of log fields;

organizing the plurality of log fields into a hierarchical data structure based on the most representative layout; and

assigning field identifiers to the plurality of log fields in the hierarchical data structure based on a corresponding hierarchical layer in the hierarchical data structure.

2. The method of claim 1, wherein the identifying pattern fields comprises:

initializing a hash table with an index key and an object set; and

populating the hash table using a value from a plurality of log fields as the index key and a field identifier corresponding the one of the plurality of log fields as the object set.

3. The method of claim 1, wherein the generating automata model comprises:

initializing a hash table with an index key and an object set; and

populating the hash table using a value from one of the pattern fields as the index key and a time stamp corresponding to the value as the object set.

4. The method of claim 3, wherein the generating automata model further comprises:

ordering the hash table firstly based on values in the index key and then secondly based on the time stamp values in the object set; and

generating the automata model correlating to all the hash tables entries for a unique pattern field value with a begin entry correlating to the hash table entry with an earliest time stamp as the object set, an end entry correlating to the hash table entry with a latest time stamp as the object set, and intermediate entries for all other hash table entries for the unique pattern field value.

6. The method of claim 1, wherein controlling the anomaly-initiating one of the plurality of network devices includes isolating the anomaly-initiating one of the plurality of network devices.

5. A method performed in a network having network devices, including computers, that generate heterogeneous logs which include a plurality of event sequences, the method comprising:

identifying, by a processor from the heterogeneous logs, pattern fields comprised of a plurality of event identifiers;

generating, by the processor, an automata model by profiling event behaviors of the plurality of event sequences, the plurality of event sequences grouped in the automata model by combinations of one or more pattern fields and one or more event identifiers from among the plurality of event identifiers, wherein for a given combination, the one or more event identifiers therein must be respectively comprised in a same one of the one or more pattern fields with which it is combined;

detecting, by the processor, an anomaly in one of the plurality of event sequences using the automata model; and

controlling, by the processor, an anomaly-initiating one of the network devices based on the anomaly,

wherein the detecting the anomaly comprises initializing a hash table for active event automata instances, performing a log grouping based on an identifier content process, and performing an event automata matching process.

7. A computer program product for automata model formation for a network having a plurality of network devices that generate heterogeneous logs which include a plurality of event sequences, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

identifying, by a processor from the heterogeneous logs, pattern fields comprised of a plurality of event identifiers;

generating, by the processor, an automata model by profiling event behaviors of the plurality of event sequences, the plurality of event sequences in the automata model grouped by combinations of one or more pattern fields and one or more event identifiers from among the plurality of event identifiers, wherein for a given combination, the one or more event identifiers therein must be respectively comprised in a same one of the one or more pattern fields with which it is combined;

detecting, by the processor, an anomaly in one of the plurality of event sequences using the automata model; and

controlling, by the processor, an anomaly-initiating one of the network devices based on the anomaly;

wherein the detecting the anomaly comprises initializing a hash table for active event automata instances, performing a log grouping based on an identifier content process, and performing an event automata matching process.

8. The computer program product of claim 7, wherein the identifying pattern fields comprises:

performing a tokenization process on the heterogeneous logs to generate tokens;

performing a log similarity process on the heterogeneous logs based on the tokens to identify log similarities amongst the heterogeneous logs; and

clustering the heterogeneous logs based on the log similarities to obtain clustered heterogeneous logs.

9. The computer program product of claim 8, wherein the identifying pattern fields further comprises:

aligning the clustered heterogeneous logs to preserve unknown layouts;

discovering a log motif to find a most representative layout of the unknown layouts and a plurality of log fields;

organizing the plurality of log fields into a hierarchical data structure based on the most representative layout; and

assigning field identifiers to the plurality of log fields in the hierarchical data structure based on a corresponding hierarchical layer in the hierarchical data structure.

10. The computer program product of claim 7, wherein the identifying pattern fields comprises:

initializing a hash table with an index key and an object set; and

populating the hash table using a value from a plurality of log fields as the index key and a field identifier corresponding to the one of the plurality of log fields as the object set.

11. The computer program product of claim 7, wherein the generating automata model comprises:

initializing a hash table with an index key and an object set; and

populating the hash table using a value from one of the pattern fields as the index key and a time stamp corresponding to the value as the object set.

12. The computer program product of claim 11, wherein the generating automata model further comprises:

ordering the hash table firstly based on values in the index key and then secondly based on the time stamp values in the object set; and

generating the automata model correlating to all the hash tables entries for a unique pattern field value with a begin entry correlating to the hash table entry with an earliest time stamp as the object set, an end entry correlating to the hash table entry with a latest time stamp as the object set, and intermediate entries for all other hash table entries for the unique pattern field value.