Field content based pattern generation for heterogeneous logs

• US 10,678,669 B2
• Filed: 04/18/2018
• Issued: 06/09/2020
• Est. Priority Date: 04/21/2017
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Debnath, Biplob, Zhang, Hui
• Primary Examiner(s): Mehrmanesh, Elmira
• Application Number:
  US15/956,381
• Publication Number:
  US 20180307576A1
• Time in Patent Office: 783 Days
• CPC Class Codes: G06F 11/0775 Content or structure details of the error report, e.g. specific table structure, specific error fields
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 2201/86 Event-based monitoring

Abstract

A system and method are provided for pattern discovery in input heterogeneous logs having unstructured text content and one or more fields. The system includes a memory. The system further includes a processor in communication with the memory. The processor runs program code to preprocess the input heterogeneous logs to obtain pre-processed logs by splitting the input heterogeneous logs into tokens. The processor runs program code to generate seed patterns from the preprocessed logs. The processor runs program code to generate final patterns by specializing a selected set of fields in each of the seed patterns to generate a final pattern set.

Claims

1. A system for pattern discovery in input heterogeneous logs having unstructured text content and one or more fields, the system comprising:

a memory; and

a processor in communication with the memory, wherein the processor runs program code to;

preprocess the input heterogeneous logs to obtain pre-processed logs by splitting the input heterogeneous logs into tokens;

generate seed patterns from the preprocessed logs; and

generate final patterns by specializing a selected set of fields in each of the seed patterns to generate a final pattern set;

wherein the processor generates the seed patterns by running program code to;

identify semantics of the tokens by assigning one of a plurality of semantic datatypes to the tokens based on Regular Expression rules;

generate seed-pattern signatures, wherein a seed-pattern signature is generated for each of the heterogeneous input logs by position-wise concatenating the semantic datatypes of the tokens therein with spaces; and

identify unique seed-pattern signatures from the seed-pattern signatures using an index, wherein each index entry includes the seed-pattern signature as an index key and associated metadata obtained as a counter value as an index value;

wherein the processor generates the seed patterns by running code to;

search the index for a given seed-pattern signature;

discard the given seed-pattern signature responsive to a matching one being found in the index and increasing the counter value; and

add the given seed-pattern signature to a database of seed-pattern signatures responsive to an absence of the matching one in the index.

2. A system for pattern discovery in input heterogeneous logs having unstructured text content and one or more fields, the system comprising:

a memory; and

a processor in communication with the memory, wherein the processor runs program code to;

preprocess the input heterogeneous logs to obtain pre-processed logs by splitting the input heterogeneous logs into tokens;

generate seed patterns from the preprocessed logs; and

generate final patterns by specializing a selected set of fields in each of the seed patterns to generate a final pattern set;

wherein the processor generates the seed patterns by running program code to;

identify semantics of the tokens by assigning one of a plurality of semantic datatypes to the tokens based on Regular Expression rules;

generate seed-pattern signatures, wherein a seed-pattern signature is generated for each of the heterogeneous input logs by position-wise concatenating the semantic datatypes of the tokens therein with spaces; and

identify unique seed-pattern signatures from the seed-pattern signatures using an index, wherein each index entry includes the seed-pattern signature as an index key and associated metadata obtained as a counter value as an index value;

wherein the processor generates the seed patterns by running code to generate a single seed-pattern for every seed-pattern signature in the index.

3. A system for pattern discovery in input heterogeneous logs having unstructured text content and one or more fields, the system comprising:

a memory; and

a processor in communication with the memory, wherein the processor runs program code to;

preprocess the input heterogeneous logs to obtain pre-processed logs by splitting the input heterogeneous logs into tokens;

generate seed patterns from the preprocessed logs; and

generate final patterns by specializing a selected set of fields in each of the seed patterns to generate a final pattern set;

wherein the processor generates the final patterns by running the program code to parse the preprocessed logs using the seed patterns to obtain parsed logs.

4. The system of claim 3, wherein the processor generates the final patterns by running code to (i) profile the parsed logs to obtain various statistics and (ii) select the set of fields in each of the seed patterns to generate the final pattern set based on the profile results.

5. A system for pattern discovery in input heterogeneous logs having unstructured text content and one or more fields, the system comprising:

a memory; and

a processor in communication with the memory, wherein the processor runs program code to;

preprocess the input heterogeneous logs to obtain pre-processed logs by splitting the input heterogeneous logs into tokens;

generate seed patterns from the preprocessed logs; and

generate final patterns by specializing a selected set of fields in each of the seed patterns to generate a final pattern set;

wherein the processor generates the final patterns from the seed patterns by running program code to select from among a plurality of pattern specializing settings selected from the group consisting of a low setting, a medium setting, and a high setting.

6. The system of claim 5, wherein different ones of the plurality of pattern specializing settings specialize different ones of the fields in each of the seed patterns to generate the final pattern set.

7. A system for pattern discovery in input heterogeneous logs having unstructured text content and one or more fields, the system comprising:

a memory; and

a processor in communication with the memory, wherein the processor runs program code to;

preprocess the input heterogeneous logs to obtain pre-processed logs by splitting the input heterogeneous logs into tokens;

generate seed patterns from the preprocessed logs; and

generate final patterns by specializing a selected set of fields in each of the seed patterns to generate a final pattern set;

wherein multiple ones of the tokens of a given field are concatenated using a specialized connector configured such that the concatenated multiples ones of the tokens are processed as a single token by a pattern generator used by the processor to generate the seed patterns.

8. A system for pattern discovery in input heterogeneous logs having unstructured text content and one or more fields, the system comprising:

a memory; and

a processor in communication with the memory, wherein the processor runs program code to;

preprocess the input heterogeneous logs to obtain pre-processed logs by splitting the input heterogeneous logs into tokens;

generate seed patterns from the preprocessed logs; and

generate final patterns by specializing a selected set of fields in each of the seed patterns to generate a final pattern set;

wherein the final patterns are generated as GROK patterns having a form that includes a syntax component and a semantic component, the syntax component denoting a pattern name to Regular Expressions text matching methodology and the semantic component denoting an identifier for a Regular Expressions text being matched.