Customized Reporting and Mining of Event Data

• US 20080172409A1
• Filed: 01/12/2007
• Published: 07/17/2008
• Est. Priority Date: 01/12/2007
• Status: Active Grant

• Current Assignee: Cloud Software Group
• Original Assignee: Loglogic (Cloud Software Group)
• Inventors: Galitsky, Boris, Liu, Minjun, Zhen, Jian L., Botros, Sherif
• Granted Patent: US 7,925,678 B2
• Time in Patent Office: Days
• US Class Current: 707/103R
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

Event data (e.g., log messages) are represented as sets of attribute/value pairs. An index maps each attribute/value pair or attribute/value tuple to a pointer that points to event data which contains the attribute/value pair or attribute/value tuple. An attribute co-occurrence map or matrix can be generated that includes attribute names that co-occur together. Queries and custom reports can be generated by projecting event data into one or more attributes or attribute/value pairs, and then determining statistics on other attributes using a combination of the inverted index, the attribute co-occurrence map or matrix, operations on sets and/or math and statistical functions.

Claims

1. A method, comprising:

receiving event data;

transforming the event data into attribute/value pairs;

generating an index operable for mapping the attribute/value pairs to a pointer that points to event data which contains the attribute/value pair; and

generating an attribute co-occurrence map including attribute names that co-occur together in the event data.

2. The method of claim 1, further comprising:

receiving a request for a report; and

responsive to the request, generating a report based on two or more related attribute/value pairs.

3. The method of claim 2, wherein generating a report further comprises:

projecting the event data into one or more attributes or attribute/value pairs; and

determining statistics on other attributes using a combination of the index and the attribute co-occurrence map.

4. The method of claim 1, wherein the statistics are determined using operations on sets of attribute/value pairs.

5. The method of claim 1, wherein the statistics are determined using statistical functions.

6. The method of claim 1, wherein the index maps an attribute/value tuple to a pointer that points to event data which contains the attribute/value tuple.

7. The method of claim 6, further comprising:

determining a count representing a number of occurrences of an attribute/value pair or attribute/value tuple for a predetermined period of time.

8. A method, comprising:

receiving a search query from one more event data collectors;

parsing the search query to determine one or more specified attributes;

retrieving values or statistics associated with the one or more specified attributes;

determining co-occurring attributes;

aggregating at least one of values or statistics associated with the co-occurring attributes; and

generating a report that includes at least one of the values or statistics.

9. A system, comprising:

a storage device operable for storing event data;

a mapper coupled to the storage device and operable for transforming the event data into attribute/value pairs;

an index coupled to the mapper and operable for mapping the attribute/value pairs to a pointer that points to event data which contains the attribute/value pair;

a reporter coupled to the index and operable for generating a report utilizing relationships among two or more related attribute/value pairs; and

an attribute co-occurrence map coupled to the reporter and operable for identifying the two or more related attribute/value pairs.

10. The system of claim 9, wherein the event data includes log messages.

11. The system of claim 9, wherein the index is operable to determine a count representing a number of occurrences of an attribute/value pair or attribute/value tuple for a predetermined period of time.

12. The system of claim 9, further comprising:

a pre-parser adapted for coupling with a number of collectors to receive event data, the pre-parser operable for identifying a source of the event data.

13. The system of claim 9, wherein the storage device is a persistent storage device configured to store event data for a user-specified period of time.

14. The system of claim 9, wherein the mapper generates attribute/value pairs by applying rules to the event data.

15. The system of claim 14, wherein the rules include a first set of rules for generating the attribute/value pairs from the event data, and a second set of rules for segmenting a space of the attribute/value pairs into regions and labeling the regions with new attribute/value pairs.

16. The system of claim 9, wherein the attribute/value pairs represent particular instances of event data.

17. The system of claim 9, wherein at least one attribute in an attribute/value pair is an action result.

18. The system of claim 9, wherein at least one attribute in an attribute/value pair is a user name.

19. The system of claim 9, wherein at least one attribute is user-definable.

20. The system of claim 9, wherein the attribute co-occurrence map comprises a two-dimensional matrix having rows and columns, each row and column representing a different attribute, the matrix including a value at each intersection of a row and column of the matrix, the value for indicating a relationship between the attributes represented by the intersecting row and column.