System and method for analysis and management of logs and events

• US 7,895,167 B2
• Filed: 02/16/2006
• Issued: 02/22/2011
• Est. Priority Date: 02/16/2005
• Status: Active Grant

• Current Assignee: Xpolog, Ltd.
• Original Assignee: Xpolog, Ltd.
• Inventors: Saguy, Amir, Berg, Gal, Koschitzky, Omry, Koschitzky, Haim
• Primary Examiner(s): Vital; Pierre M
• Assistant Examiner(s): Vo; Truong V
• Application Number:
  US11/354,930
• Publication Number:
  US 20060184529A1
• Time in Patent Office: 1,832 Days
• Field of Search: 707/672
• US Class Current: 707/672
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0784 Routing of error reports, e.g. with a specific transmission path or data flow
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36

Abstract

A log record analyzing system for monitoring log records from at least one computerized system. The log record analyzing system comprises a pattern repository that stores a plurality of pattern object records of different grammar types and a parsing engine which is adapted to receive a raw log data input. The parsing engine facilitates the matching between the raw log data input and at least one of the pattern object records. The parsing engine outputs parsed data according to the matching.

Claims

1. A log record analyzing system for monitoring log records from external computerized systems, the external computerized systems using respectively initially undefined grammar types for log records, said log record analyzing system comprising:

a processor, the processor being hardware and configured with;

a pattern repository configured to store a plurality of pattern object records, each of said stored pattern object records being of a respectively different log record grammar type, thereby to provide within said system a plurality of defined grammar types, incoming log records being matchable with said pattern records to find a matching structure, thereby to parse said incoming log records of undefined grammar type according to respectively matched structures, said matching thereby defining grammar types for parsing of said incoming log records; and

a parsing engine associated with said electronic pattern repository, comprising;

a raw log data input for receiving raw log data from said computerized system, said raw log data being of said undefined grammar type,

a matching unit associated with said input for matching between said raw log data input and successive ones of said pattern object records to find one of said pattern object records having a structure most closely matching said raw log data irrespective of said matching being an exact match, selecting said pattern object record having said most closely matching structure and parsing said raw log data of undefined grammar type using said selected pattern object record and said matching structure to produce a parsed structured version of said raw log data of initially undefined grammar type; and

an output for outputting said parsed structured version of said raw log data of initially undefined grammar type, said parsed structured version thereby being rendered suitable for said monitoring, said closest matching being to ensure that said parsed structured version is provided and said monitoring enabled even when no exactly matching pattern object record is found, the system further comprising an automatic parsed data builder configured to identify the grammar of said raw log data input, said automatic parsed data builder being configured to output a pattern object according to said identified grammar, storing said pattern object in said pattern repository.

2. The log record analyzing system of claim 1, wherein said raw log data input consists of at least one of the following members:

textual log files, XML files, database files, communication information unit carriages, Java Message Service (JMS) packet files, and Enterprise Application Integration (EAI) packet files.

3. The log record analyzing system of claim 1, wherein said raw log data input consists of at least one of the following data types:

semi-structured data, unstructured data, and structured data.

4. The log record analyzing system of claim 1, wherein said raw log data comprises at least one of the following entries:

multiple line entry and single line entry.

5. The log record analyzing system of claim 1, wherein said raw log data input comprises a plurality of grammar types.

6. The log record analyzing system of claim 1, further comprising a Complex Event Processing (CEP) module configured to receive said raw log data from at least one computerized system, said CEP module being configured to forward said received raw log data to said parsing engine, wherein said forwarding is done according to a set of predetermined rules.

7. The log record analyzing system of claim 6, wherein said CEP module is configured to be connected to a transaction database operative for storing said raw log data, wherein said CEP module is configured to further transmit said received raw log data to said transaction database.

8. The log record analyzing system of claim 6, wherein said CEP module is configured to be connected to a user interface device, wherein said user interface device is configured to transmit said set of predetermined rules to said CEP module.

9. The log record analyzing system of claim 6, wherein said set of predetermined rules comprises at least one of the following rules:

static rule, dynamic rule, deterministic rule, statistical rule, event driven rule, and time and date based rule.

10. The log record analyzing system of claim 6, wherein said set of predetermined rules comprises a rule that requires the existence of a predefined pattern in said raw log data.

11. The log record analyzing system of claim 1, wherein said parsed structured data is a list containing at least one array of objects, wherein said objects comprise parsed raw log data.

12. The log record analyzing system of claim 1, further comprising a parsed data viewer, wherein said parsed data viewer is configured to receive said parsed data and wherein said viewer is adapted to graphically display said received parsed data.

13. The log record analyzing system of claim 1, further comprising a log record builder, wherein said log record builder is configured to receive said parsed structured data;

wherein said log record builder is configured to output at least one resultant log record according to a set of predetermined rules, said output being based upon said raw log data input and said parsed structured data.

14. The log record analyzing system of claim 13, wherein said log record builder is configured to be connected to a user interface device, said user interface device operable for transmitting said set of predetermined rules to said log record builder.

15. The log record analyzing system of claim 13, further comprising a log record viewer operative to display said at least one resultant log record.

16. The log record analyzing system of claim 13, further comprising an indexing module, said indexing module configured to receive said at least one resultant log record, and, based thereupon, to output at least one indexed resultant log record.

17. The log record analyzing system of claim 16, further comprising a search module, wherein said search module is configured to search said at least one indexed resultant log record according to at least one search definition, said search module being configured to output a matching list consisting of at least one of said resultant log records.

18. The log record analyzing system of claim 17, further comprising a user interface, said user interface facilitating the input of said at least one search definition.