Method and system for implementing a log parser in a log analytics system

• US 10,366,096 B2
• Filed: 04/01/2016
• Issued: 07/30/2019
• Est. Priority Date: 04/03/2015
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Ferrar, Gregory Michael
• Primary Examiner(s): Wilson, Kimberly L
• Application Number:
  US15/089,180
• Publication Number:
  US 20160292263A1
• Time in Patent Office: 1,215 Days
• Field of Search: 707811
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/0775 Content or structure details of the error report, e.g. specific table structure, specific error fields
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/3086 where the reporting involves the use of self describing data formats, i.e. metadata, markup languages, human readable formats
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 16/84 Mapping; Conversion
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 40/16 Automatic learning of transformation rules, e.g. from examples
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 9/44505 Configuring for program initiating, e.g. using registry, configuration files
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 41/145 involving simulating, designing, planning or modelling of a network
• CPC Class Codes: H04L 41/5074 Handling of user complaints or trouble tickets
• CPC Class Codes: H04L 43/04 : Processing captured monitoring data, e.g. for logfile generation

Abstract

Disclosed is a system, method, and computer program product for implementing a log analytics method and system that can configure, collect, and analyze log records in an efficient manner. An improved approach has been described to automatically generate a log parser by analysis of the line content of a log. In addition, an efficient approach has been described to extract key-value content from the log content.

Claims

1. A method comprising:

generating a mapping structure that maps a first plurality of elements of a first entry in a log to corresponding element types;

analyzing a second plurality of elements of a second entry in the log, relative to the mapping structure, to identify at least;

a common element that is identical in the first plurality of elements and the second plurality of elements, and

a first variable element that varies between the first plurality of elements and the second plurality of elements;

generating a regular expression comprising at least;

the common element, and

a first matching pattern that (a) restricts the first variable element to a first data type, from a plurality of possible data types, and (b) matches the first variable element in both the first plurality of elements and the second plurality of elements; and

generating a log parser configured to parse logs based on the regular expression, wherein the method is performed by at least one device comprising a processor.

2. The method of claim 1, wherein analyzing the second plurality of elements relative to the mapping structure comprises:

traversing the second plurality of elements;

while traversing the second plurality of elements;

determining a first common character that is identical between the second plurality of elements and the mapping structure,

determining a second common character that is identical between the second plurality of elements and the mapping structure, and

determining that one or more intervening characters, between the first common character and the second common character, does not match between the second plurality of elements and the mapping structure; and

marking a range of characters, encompassing the one or more intervening characters, as the first variable element.

3. The method of claim 1, wherein the plurality of possible data types comprises one or more of a string type, an integer type, an alphabetic character type, or a field rule type, wherein the field rule type corresponds to a sequence of elements defined by a rule.

4. The method of claim 1, further comprising:

grouping together contents of a plurality of lines of the log, as the second entry for analysis relative to the mapping structure.

5. The method of claim 4, wherein grouping together the contents of the plurality of lines comprises forming a single line comprising the contents.

6. The method of claim 1, further comprising:

identifying that the common element is a delimiter within the log, at least by;

identifying a plurality of common elements in the first entry and the second entry;

scoring the plurality of common elements, based at least in part on respective positions of the plurality of common elements and one or more weighting factors; and

selecting the common element as the delimiter, based at least in part on the scoring.

7. The method of claim 1, further comprising:

identifying a second variable element that varies between the first plurality of elements and the second plurality of elements,

wherein the regular expression further comprises a second matching pattern that (a) restricts the second variable element to a second data type, from the plurality of possible data types, and (b) matches the second variable element in both the first plurality of elements and the second plurality of elements,

wherein the first data type is different from the second data type.

8. A non-transitory computer readable medium having stored thereon instructions which, when executed by one or more hardware processors, cause performance of operations comprising:

generating a mapping structure that maps a first plurality of elements of a first entry in a log to corresponding element types;

analyzing a second plurality of elements of a second entry in the log, relative to the mapping structure, to identify at least;

a common element that is identical in the first plurality of elements and the second plurality of elements, and

a first variable element that varies between the first plurality of elements and the second plurality of elements;

generating a regular expression comprising at least;

the common element, and

a first matching pattern that (a) restricts the first variable element to a first data type, from a plurality of possible data types, and (b) matches the first variable element in both the first plurality of elements and the second plurality of elements; and

generating a log parser configured to parse logs based on the regular expression,

wherein the method is performed by at least one device comprising a processor.

9. The non-transitory computer readable medium of claim 8, wherein analyzing the second plurality of elements relative to the mapping structure comprises:

traversing the second plurality of elements;

while traversing the second plurality of elements;

determining a first common character that is identical between the second plurality of elements and the mapping structure,

determining a second common character that is identical between the second plurality of elements and the mapping structure, and

determining that one or more intervening characters, between the first common character and the second common character, does not match between the second plurality of elements and the mapping structure; and

marking a range of characters, encompassing the one or more intervening characters, as the first variable element.

10. The non-transitory computer readable medium of claim 8, wherein the plurality of possible data types comprises one or more of a string type, an integer type, an alphabetic character type, or a field rule type, wherein the field rule type corresponds to a sequence of elements defined by a rule.

11. The non-transitory computer readable medium of claim 8, the operations further comprising:

grouping together contents of a plurality of lines of the log, as the second entry for analysis relative to the mapping structure.

12. The non-transitory computer readable medium of claim 11, wherein grouping together the contents of the plurality of lines comprises forming a single line comprising the contents.

13. The non-transitory computer readable medium of claim 8, the operations further comprising:

determining that the common element is a delimiter within the log, at least by;

identifying a plurality of common elements in the first entry and the second entry;

scoring the plurality of common elements, based at least in part on respective positions of the plurality of common elements and one or more weighting factors; and

selecting the common element as the delimiter, based at least in part on the scoring.

14. The non-transitory computer readable medium of claim 8, the operations further comprising:

identifying a second variable element that varies between the first plurality of elements and the second plurality of elements,

wherein the regular expression further comprises a second matching pattern that (a) restricts the second variable element to a second data type, from the plurality of possible data types, and (b) matches the second variable element in both the first plurality of elements and the second plurality of elements,

wherein the first data type is different from the second data type.

15. A system, comprising:

at least one device including a hardware processor;

the system being configured to perform operations comprising;

generating a mapping structure that maps a first plurality of elements of a first entry in a log to corresponding element types;

analyzing a second plurality of elements of a second entry in the log, relative to the mapping structure, to identify at least;

a common element that is identical in the first plurality of elements and the second plurality of elements, and

a first variable element that varies between the first plurality of elements and the second plurality of elements;

generating a regular expression comprising at least;

the common element, and

a first matching pattern that (a) restricts the first variable element to a first data type, from a plurality of possible data types, and (b) matches the first variable element in both the first plurality of elements and the second plurality of elements; and

generating a log parser configured to parse logs based on the regular expression,

wherein the method is performed by at least one device comprising a processor.

16. The system of claim 15, wherein analyzing the second plurality of elements relative to the mapping structure comprises:

traversing the second plurality of elements;

while traversing the second plurality of elements;

determining a first common character that is identical between the second plurality of elements and the mapping structure,

determining a second common character that is identical between the second plurality of elements and the mapping structure, and

determining that one or more intervening characters, between the first common character and the second common character, does not match between the second plurality of elements and the mapping structure; and

marking a range of characters, encompassing the one or more intervening characters, as the first variable element.

17. The system of claim 15, wherein the plurality of possible data types comprises one or more of a string type, an integer type, an alphabetic character type, or a field rule type, wherein the field rule type corresponds to a sequence of elements defined by a rule.

18. The system of claim 15, the operations further comprising:

grouping together contents of a plurality of lines of the log, as the second entry for analysis relative to the mapping structure.

19. The system of claim 15, the operations further comprising:

identifying that the common element is a delimiter within the log, at least by;

identifying a plurality of common elements in the first entry and the second entry;

scoring the plurality of common elements, based at least in part on respective positions of the plurality of common elements and one or more weighting factors; and

selecting the common element as the delimiter, based at least in part on the scoring.

20. The system of claim 15, the operations further comprising:

identifying a second variable element that varies between the first plurality of elements and the second plurality of elements,

wherein the regular expression further comprises a second matching pattern that (a) restricts the second variable element to a second data type, from the plurality of possible data types, and (b) matches the second variable element in both the first plurality of elements and the second plurality of elements,

wherein the first data type is different from the second data type.