Customized reporting and mining of event data

• US 7,925,678 B2
• Filed: 01/12/2007
• Issued: 04/12/2011
• Est. Priority Date: 01/12/2007
• Status: Active Grant

• Current Assignee: Cloud Software Group
• Original Assignee: Loglogic (Cloud Software Group)
• Inventors: Galitsky, Boris, Liu, Minjun, Zhen, Jian L., Botros, Sherif
• Primary Examiner(s): LeRoux; Etienne P
• Assistant Examiner(s): Pyo; Monica M
• Application Number:
  US11/623,010
• Publication Number:
  US 20080172409A1
• Time in Patent Office: 1,551 Days
• Field of Search: 707/10, 707/101, 707/102, 707/803, 707999102-999104, 707/103.X
• US Class Current: 707/803
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

Event data (e.g., log messages) are represented as sets of attribute/value pairs. An index maps each attribute/value pair or attribute/value tuple to a pointer that points to event data which contains the attribute/value pair or attribute/value tuple. An attribute co-occurrence map or matrix can be generated that includes attribute names that co-occur together. Queries and custom reports can be generated by projecting event data into one or more attributes or attribute/value pairs, and then determining statistics on other attributes using a combination of the inverted index, the attribute co-occurrence map or matrix, operations on sets and/or math and statistical functions.

Claims

1. A computer-implemented method, comprising:

receiving event data in a log file;

transforming the event data into attribute/value pairs;

generating an index mapping at least one of the attribute/value pairs to a reference pointer that references an instance of event data in the log file, the instance of event data in the log file including a textual message from which the attribute/value pair was transformed;

generating an attribute co-occurrence data structure, the attribute co-occurrence data structure identifying a relationship between a first attribute in the attribute/value pairs and a second attribute in the attribute/value pairs, where existence of the relationship between the first attribute and the second attribute indicates that reports for finding a textual message including both the first attribute and the second attribute are allowable; and

in response to a query including the first attribute, generating a report based on the attribute co-occurrence data structure and the attribute/value pairs in the index, including;

identifying the second attribute based on the relationship identified in the attribute co-occurrence data structure;

identifying an attribute/value pair that includes the second attribute; and

generating the report, including providing access to the event data in the log file using a reference pointer that is mapped to the identified attribute/value pair in the index,

where the method is performed by one or more processors.

2. The method of claim 1, where identifying the relationship between the first attribute and the second attribute includes identifying a sender-recipient relationship between the first attribute and the second attribute according to an email message.

3. The method of claim 1, wherein generating the report further comprises:

projecting the event data into one or more attributes or attribute/value pairs; and

determining statistics on other attributes using a combination of the index and the attribute co-occurrence data structure.

4. The method of claim 3, wherein the statistics are determined using operations on sets of attribute/value pairs.

5. The method of claim 3, wherein the statistics are determined using statistical functions.

6. The method of claim 1, wherein the index maps an attribute/value tuple to a reference pointer that references an instance of event data that contains the attribute/value tuple.

7. The method of claim 6, further comprising:

determining a count representing a number of occurrences of an attribute/value pair or attribute/value tuple for a predetermined period of time.

8. A system, comprising:

a storage device operable for storing event data in a log file; and

one or more data processing devices configured to perform operations comprising;

transforming the event data into attribute/value pairs;

generating an index mapping at least one of the attribute/value pairs to a reference pointer that references an instance of event data in the log file, the instance of event data in the log file including a textual message from which the attribute/value pair was transformed;

generating an attribute co-occurrence data structure, the attribute co-occurrence data structure identifying a relationship between a first attribute in the attribute/value pairs and a second attribute in the attribute/value pairs, where existence of the relationship between the first attribute and the second attribute indicates that reports for finding a textual message including both the first attribute and the second attribute are allowable; and

in response to a query including the first attribute, generating a report based on the attribute co-occurrence data structure and the attribute/value pairs in the index, including;

identifying the second attribute based on the relationship identified in the attribute co-occurrence data structure;

identifying an attribute/value pair that includes the second attribute; and

generating the report, including providing access to the event data in the log file using a reference pointer that is mapped to the identified attribute/value pair in the index.

9. The system of claim 8, wherein the event data includes log messages.

10. The system of claim 8, wherein the index is operable to determine a count representing a number of occurrences of an attribute/value pair or attribute/value tuple for a predetermined period of time.

11. The system of claim 8, further comprising:

a pre-parser adapted for coupling with a number of collectors to receive the event data, the pre-parser operable for identifying a source of the event data.

12. The system of claim 8, wherein the storage device is a persistent storage device configured to store the event data for a user-specified period of time.

13. The system of claim 8, wherein transforming the event data into attribute/value pairs includes generating the attribute/value pairs by applying rules to the event data.

14. The system of claim 13, wherein the rules include a first set of rules for generating the attribute/value pairs from the event data, and a second set of rules for segmenting a space of the attribute/value pairs into regions and labeling the regions with new attribute/value pairs.

15. The system of claim 8, wherein the attribute/value pairs represent particular instances of event data.

16. The system of claim 8, wherein at least one attribute in an attribute/value pair is an action result.

17. The system of claim 8, wherein at least one attribute in an attribute/value pair is a user name.

18. The system of claim 8, wherein at least one attribute is user-definable.

19. The system of claim 8, wherein the attribute co-occurrence data structure comprises a two-dimensional matrix having rows and columns, each row and column representing a different attribute, the matrix including a value at each intersection of a row and column of the matrix, the value for indicating a relationship between the attributes represented by the intersecting row and column.