SEARCHING FOR ASSOCIATED EVENTS IN LOG DATA

• US 20090089252A1
• Filed: 10/02/2007
• Published: 04/02/2009
• Est. Priority Date: 10/02/2007
• Status: Active Grant

• Current Assignee: Cloud Software Group
• Original Assignee: TIBCO Software Incorporated (Cloud Software Group)
• Inventors: Galitsky, Boris, Botros, Sherif
• Granted Patent: US 8,306,967 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/332 Query formulation
• CPC Class Codes: G06F 16/3335 Syntactic pre-processing, e.g. stopword elimination, stemming

Abstract

To retrieve a sequence of associated events in log data, a request expression is parsed to retrieve types of dependencies between events which are searched, and the constraints (e.g., keywords) which characterize each event. Based on the parsing results, query components can be formed, expressing the constraints for individual events and interrelations (e.g., time spans) between events. A resultant span query comprising the query components can then be run against an index of events, which encodes a mutual location of associated events in storage.

Claims

1. A method of searching for associated events in log data, comprising:

storing one or more events as log messages;

creating an index of terms in the log messages in a form so that one or more events can be retrieved in response to a first query;

parsing the first query;

running the first query;

obtaining results from running of first query;

forming a second query for associated events based on the results; and

performing the second query on the index to determine associated events.

2. The method of claim 1, further comprising:

grouping the determined associated events by attributes.

3. The method of claim 1, wherein the second query is selected from a group of query types consisting of a sequence of events query, a multiple restriction query, and a causation query.

4. The method of claim 3, wherein forming the sequence of events query comprises:

building one or more queries for individual log messages; and

merging the queries for individual log messages as a span query over multiple log messages.

5. The method of claim 4, wherein performing the sequence of events query comprises:

performing the queries for individual log messages; and

determining a time span between results of the queries for individual log messages; and

verifying that the log messages occur within the determined time span.

6. The method of claim 3, wherein forming the multiple restriction query comprises:

determining a first query for an intermediate event; and

determining a second query for a final event.

9. The method of claim 1, wherein the second query includes multiple query types and at least one query type includes at least one intermediate query.

7. A method of searching for associated events in log data, comprising:

processing a first query to identify constant and variable components;

forming a second query by merging the constant and variable components;

performing the second query to identify merged events to be explained;

retrieving events preceding the identified merged events within a first time span;

performing a third query based on the constant component to identify constant events to be explained;

retrieving events preceding the identified constant events within a second time span;

building a first set of intersections for identified merged events;

building a second set of intersections for identified constant events; and

determining differences between intersections in the first and second sets of intersections.

8. The method of claim 7, further comprising:

outputting a list of differences as possible answers to the first query.

10. A system for identifying a sequence of events in event data, comprising:

a storage device operable to index event data for retrieval; and

a processor coupled to the storage device and operable to interpret an expression specified by a user to build a query for a sequence of events, and to identify a sequence of events satisfying conditions included in the query.

11. The system of claim 10, where the event data includes log messages.

12. A system comprising:

a storage device operable for storing one or more events as log messages; and

a processor coupled to the storage device and operable for;

creating an index of terms in the log messages in a form so that one or more events can be retrieved in response to a first query;

parsing the first query;

running the first query;

obtaining results from running of first query;

forming a second query for associated events based on the results; and

performing the second query on the index to determine associated events.

13. The system of claim 12, further comprising:

grouping the determined associated events by attributes.

14. The system of claim 12, wherein the second query is selected from a group of query types consisting of a sequence of events query, a multiple restriction query, and a causation query.

15. The system of claim 14, wherein forming the sequence of events query comprises:

building one or more queries for individual log messages; and

merging the queries for individual log messages as a span query over multiple log messages.

16. The system of claim 15, wherein performing the sequence of events query comprises:

performing the queries for individual log messages; and

determining a time span between results of the queries for individual log messages; and

verifying that the log messages occur within the determined time span.

17. The system of claim 15, wherein performing the sequence of events query comprises:

performing a single query which includes a time span as a constraint.

18. The system of claim 14, wherein forming the multiple restriction query comprises:

determining a third query for an intermediate event; and

determining a fourth query for a final event.

19. A system comprising:

a storage device operable for storing one or more events as log messages; and

a processor coupled to the storage device and operable for;

processing a first query to identify constant and variable components;

forming a second query by merging the constant and variable components;

performing the second query to identify merged events to be explained;

retrieving events preceding the identified merged events within a first time span;

performing a third query based on the constant component to identify constant events to be explained;

retrieving events preceding the identified constant events within a second time span;

building a first set of intersections for identified merged events;

building a second set of intersections for identified constant events; and

determining differences between intersections in the first and second sets of intersections.

20. The system of claim 19, further comprising:

outputting a list of differences as possible answers to the first query.

21. A computer-readable medium having instructions stored thereon, which, when executed by a processor, causes the processor to perform operations comprising:

storing one or more events as log messages;

creating an index of terms in the log messages in a form so that one or more events can be retrieved in response to a first query;

parsing the first query;

running the first query;

obtaining results from running of first query;

forming a second query for associated events based on the results; and

performing the second query on the index to determine associated events.

22. The computer-readable medium of claim 21, further comprising:

grouping the determined associated events by attributes.

23. The computer-readable medium of claim 21, wherein the second query is selected from a group of query types consisting of a sequence of events query, a multiple restriction query, and a causation query.

24. A system comprising:

means for storing one or more events as log messages;

means for creating an index of terms in the log messages in a form so that one or more events can be retrieved in response to a first query;

means for parsing the first query;

means for running the first query;

means for obtaining results from running of first query;

means for forming a second query for associated events based on the results; and

means for performing the second query on the index to determine associated events.

25. The system of claim 24, wherein the second query is selected from a group of query types consisting of a sequence of events query, a multiple restriction query, and a causation query.