System and method for analysis and management of logs and events

• US 20060184529A1
• Filed: 02/16/2006
• Published: 08/17/2006
• Est. Priority Date: 02/16/2005
• Status: Active Grant

• Current Assignee: Xpolog, Ltd.
• Original Assignee: Xpolog, Ltd.
• Inventors: Saguy, Amir, Berg, Gal, Koschitzky, Omry, Koschitzky, Haim
• Granted Patent: US 7,895,167 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0784 Routing of error reports, e.g. with a specific transmission path or data flow
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36

Abstract

A log record analyzing system for monitoring log records from at least one computerized system. The log record analyzing system comprises a pattern repository that stores a plurality of pattern object records of different grammar types and a parsing engine which is adapted to receive a raw log data input. The parsing engine facilitates the matching between the raw log data input and at least one of the pattern object records. The parsing engine outputs parsed data according to the matching.

Claims

1. A log record analyzing system for monitoring a log record from at least one computerized system, said log record analyzing system comprising:

a pattern repository adapted to store more than one pattern object record of different grammar types; and

a parsing engine associated with said pattern repository, comprising;

a raw log data input for receiving raw log data,

a matching unit associated with said input for matching between said raw log data input and one of said pattern object records; and

an output for outputting a parsed structured version of said raw log data using a structure extracted from said matched record.

2. The log record analyzing system of claim 1, wherein said raw log data input consists of at least one of the following members:

textual log files, XML files, database files, communication information unit carriages, Java Message Service (JMS) packet files, and Enterprise Application Integration (EAI) packet files.

3. The log record analyzing system of claim 1, wherein said raw log data input consists of at least one of the following data types:

semi-structured data, unstructured data, and structured data.

4. The log record analyzing system of claim 1, wherein said raw log data comprises at least one of the following entries:

multiple line entry and single line entry.

5. The log record analyzing system of claim 1, wherein said raw log data input comprises a plurality of grammar types.

6. The log record analyzing system of claim 1, further comprising a Complex Event Processing (CEP) module adapted to receive raw log data from at least one computerized system, said CEP module being configured to forward said received raw log data to said parsing engine, wherein said forwarding is done according to a set of predetermined rules.

7. The log record analyzing system of claim 6, wherein said CEP module is adapted to be connected to a transaction database operative for storing said raw log data, wherein said CEP module further transmits said received raw log data to said transaction database.

8. The log record analyzing system of claim 6, wherein said CEP module is adapted to be connected to a user interface device, wherein said user interface device is adapted to transmit said set of predetermined rules to said CEP module.

9. The log record analyzing system of claim 6, wherein said set of predetermined rules comprises at least one of the following rules:

static rule, dynamic rule, deterministic rule, statistical rule, event driven rule, and time and date based rule.

10. The log record analyzing system of claim 6, wherein said set of predetermined rules comprises a rule that requires the existence a predefined pattern in said raw log data.

11. The log record analyzing system of claim 1, wherein said parsed structured data is a list containing at least one array of objects, wherein said objects comprise parsed raw log data.

12. The log record analyzing system of claim 1, further comprising a parsed data viewer, wherein said parsed data viewer is adapted to receive said parsed data and wherein said viewer is adapted to graphically display said received parsed data.

13. The log record analyzing system of claim 1, further comprising a log record builder, wherein said log record builder is adapted to receive said parsed structured data;

wherein said log record builder is configured to output at least one resultant log record according to a set of predetermined rules, said output being based upon said raw log data input and said parsed structured data.

14. The log record analyzing system of claim 13, wherein said log record builder is adapted to be connected to a user interface device, said user interface device operable for transmitting said set of predetermined rules to said log record builder.

15. The log record analyzing system of claim 13, further comprising a log record viewer operative to display said at least one resultant log record.

16. The log record analyzing system of claim 13, further comprising an indexing module, said indexing module adapted to receive said at least one resultant log record, and, based thereupon, to output at least one indexed resultant log record.

17. The log record analyzing system of claim 16, further comprising a search module, wherein said searched module is configured to search said at least one indexed resultant log record according to at least one search definition, said search module being adapted to output a matching list consisting of at least one of said resultant log records.

18. The log record analyzing system of claim 17, further comprising a user interface, said user interface facilitating the input of said at least one search definition.

19. The log record analyzing system of claim 1, further comprising an automatic parsed data builder adapted to identify the grammar of said raw log data input, said automatic parsed data builder being adapted to output a pattern object according to said identified grammar, storing said pattern object in said pattern repository.

20. A searching apparatus for searching specific entries in raw log data from at least one computerized system, said searching apparatus comprising:

an indexing module adapted to scan said raw log data and, based thereupon, to output indexed log data, said indexed log data comprising a plurality of term records and a plurality of corresponding position records, said position records reflecting the positions of terms in said raw log data;

an indexed log file repository operable for storing at least one of said indexed log data; and

a search module comprising;

an input for receiving a search query,

an index search unit for searching said at least one indexed log data for records that match said search query, and

an output to output a list of said matched records.

21. The searching apparatus of claim 20, wherein said search module is adapted to associate said matched records with entries of said raw log data according to said corresponding position records output, said position records based upon a list of matched entries.

22. The searching apparatus of claim 20, wherein said raw log data comprises at least one of the following data types:

unstructured data, semi-structured data, and structured data.

23. The searching apparatus of claim 20, further comprising a displaying module adapted to output a visual display of said matched records.

24. An automatic pattern recognition apparatus for identifying patterns of raw log data having different grammar types, said automatic pattern recognition apparatus comprising:

a pattern repository operable for storing at least one pattern object of different grammar patterns;

an automatic pattern recognition module comprising;

an identification module adapted to identify the grammar of said raw log data input, and comprising an input to receive at least a portion of said raw log data, and

an output unit for outputting a pattern object record generated to represent said identified grammar, and to store said respective pattern object in said pattern repository; and

a parsing engine comprising an input to receive a raw log data input, a matching unit to match between said raw log data input and at least one of said pattern object records; and

an output unit to output parsed data according to said matching.

25. The automatic pattern recognition apparatus of claim 24, wherein said automatic pattern recognition module is adapted to be connected to a user interface, said user interface being operable for receiving said pattern object from said automatic pattern recognition module and, based thereupon, displaying a visual display of said pattern object, said user interface operative for facilitating the editing of said pattern object.

26. A method for parsing log data with undefined grammar, comprising:

a) storing more than one pattern object record of different grammar types;

b) receiving at least a portion of raw log data input from at least one computerized system;

c) identifying the delimiter of said portion of raw log data'"'"'s grammar;

d) using said delimiter for generating a new pattern object representing the grammar type of said log data, said new pattern object comprising a list of terms; and

e) storing said new pattern object.

27. The method of claim 26, wherein said storing of step (e) includes merging said new pattern object with a similar pattern object record or storing said pattern object if a similar pattern object has not already being stored.

28. The method of claim 26, wherein said list of terms is arranged as a prefix tree.

29. The method of claim 26, further comprising a step between said step (d) and said step (e) of enabling users to edit said new pattern object.

30. A method for monitoring computerized systems, comprising the following steps:

a) storing more than one pattern object record of different grammar types;

b) receiving raw log data input from at least one computerized system;

c) matching the grammar type of said raw log data input with one of said pattern object record;

d) parsing said raw log data input according the grammar type of said matched pattern object record; and

e) outputting at least one unit of parsed data based upon said parsing.

31. The method of claim 30, further comprising a step of displaying said at least one unit of parsed data.

32. The method of claim 31, wherein step (a) further comprises providing an automatic pattern recognition module;

wherein said step (c) further comprises using said automatic pattern recognition module for identifying the grammar of said raw log data input and for updating the pattern of said data based thereupon.

33. The method of claim 32, further comprising:

a step between step (b) and step (c) of analyzing the grammar type of said raw log data input using said automatic pattern recognition module, outputting a new pattern object based thereupon; and

a step of storing said new pattern object.