Machine Data Web

• US 20070118491A1
• Filed: 07/24/2006
• Published: 05/24/2007
• Est. Priority Date: 07/25/2005
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Murphy, Brian, Sorkin, Stephen, Stechert, Andre, Carasso, David, Hall, Brad, Baum, Michael, Das, Robin, Swan, Erik
• Granted Patent: US 7,937,344 B2
• Time in Patent Office: Days
• US Class Current: 706/20
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24564 Applying rules; Deductive queries
• CPC Class Codes: G06F 16/24573 using data annotations, e.g. user-defined metadata
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/288 Entity relationship models
• CPC Class Codes: G06F 16/316 Indexing structures
• CPC Class Codes: G06F 16/3331 Query processing
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06F 9/54 Interprogram communication
• CPC Class Codes: G06F 9/541 via adapters, e.g. between incompatible applications
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications

Abstract

Methods and apparatus consistent with the invention provide the ability to organize and build understandings of machine data generated by a variety of information-processing environments. Machine data is a product of information-processing systems (e.g., activity logs, configuration files, messages, database records) and represents the evidence of particular events that have taken place and been recorded in raw data format. In one embodiment, machine data is turned into a machine data web by organizing machine data into events and then linking events together.

Claims

1. A method for building a machine data web from machine data, the method comprising:

organizing machine data into events; and

based on event data for the events, creating a machine data web of linked events.

2. The method of claim 1 wherein the step of organizing machine data into events comprises:

collecting machine data;

classifying the collected machine data into domains; and

on a domain by domain basis, aggregating the collected machine data into events.

3. The method of claim 1 wherein the step of creating a machine data web of events comprises:

creating dynamic links between events; and

chaining linked events to create paths.

4. The method of claim 1 further comprising:

analyzing the machine data web to generate historical information about the events.

5. The method of claim 1 wherein the machine data web is Internet-based;

data can be added to the machine data web by a command to post data to the machine data web, and data can be retrieved from the machine data web by a command to get data from the machine data web.

6. A method for organizing machine data into events comprising:

collecting machine data;

classifying the collected machine data into domains; and

on a domain by domain basis, aggregating the collected machine data into events.

7. The method of claim 6 wherein the step of classifying the collected machine data into domains comprises classifying the collected machine data into domains based on a source of the machine data.

8. The method of claim 7 wherein the domains include both general domains and specific domains.

9. The method of claim 7 wherein the step of classifying the collected machine data into domains based on a source of the machine data comprises:

creating source signatures for the domains; and

classifying the collected machine data into domains based on the source signatures.

10. The method of claim 9 wherein the step of creating source signatures for the domains comprises learning a source signature for a domain based on the machine data collected from that domain.

11. The method of claim 9 wherein the step of classifying the collected machine data into domains based on the source signatures is based on clustering.

12. The method of claim 9 wherein the step of classifying the collected machine data into domains based on the source signatures is based on performing a nearest neighbor search.

13. The method of claim 9 wherein the source signatures are based on a frequency of punctuation characters in the machine data.

14. The method of claim 9 wherein the source signatures are based on functions of punctuation in the machine data.

15. The method of claim 7 wherein the step of classifying the collected machine data into domains based on a source comprises:

converting binary machine data into textual form; and

classifying the textual form of the collected machine data.

16. The method of claim 7 wherein at least some of the events are multi-line type events.

17. The method of claim 6 wherein the step of aggregating the collected machine data into events comprises aggregating the machine data based on break rules for the domain.

18. The method of claim 17 wherein the step of aggregating the collected machine data into events comprises learning the break rules for a domain based on the machine data collected from that domain.

19. The method of claim 17 wherein the break rules have confidence levels.

20. The method of claim 6 further comprising:

extracting semantic entities from event data for the events.

21. The method of claim 6 further comprising:

segmenting event data for the events into segments.

22. The method of claim 21 wherein the step of segmenting event data into segments comprises segmenting event data based on major and minor breaks.

23. The method of claim 6 further comprising:

classifying the events into event types.

24. The method of claim 23 wherein the step of classifying the events into event types comprises:

creating event signatures for the event types; and

classifying the events into event types based on the event signatures.

25. The method of claim 24 wherein the event signatures are based on a hierarchical scheme, and the step of classifying the events into event types comprises traversing the hierarchical scheme.

26. A method for creating a machine data web of events comprising:

creating dynamic links between events; and

chaining linked events to create paths.

27. The method of claim 26 wherein the step of creating dynamic links between events comprises creating dynamic links based on a value analysis.

28. The method of claim 26 wherein the step of creating dynamic links between events comprises creating dynamic links based on a statistical analysis.

29. The method of claim 26 wherein the step of creating dynamic links between events comprises creating dynamic links based on a timing analysis.

30. The method of claim 26 wherein the step of creating dynamic links between events comprises creating dynamic links based on link hints.

31. The method of claim 26 wherein the step of creating dynamic links between events comprises:

creating a co-occurrence table of pairs of events; and

creating dynamic links between events based on the co-occurrence table.

32. The method of claim 31 wherein the step of creating dynamic links between events based on the co-occurrence table comprises creating a dynamic link between a pair of events, if the pair co-occurs more frequently than a predetermined frequency and a distance between the events in the pair is less than a predetermined distance.

33. The method of claim 26 wherein the paths comprise chains of transitive links of events.

34. A system for building a machine data web from machine data, the system comprising:

a machine data organization module for organizing machine data into events; and

a machine data understanding module for creating a machine data web of linked events based on event data for the events.

35. The system of claim 34 wherein the machine data organization module comprises:

a collection module for collecting machine data;

a classification module for classifying the collected machine data into domains; and

an aggregation module for, on a domain by domain basis, aggregating the collected machine data into events.

36. The system of claim 34 wherein the machine data understanding module comprises:

a linking module for creating dynamic links between events; and

a path construction module for chaining linked events to create paths.

37. A computer readable medium containing instructions to cause processor(s) to execute instructions to carry out a method for building a machine data web from machine data, the method comprising:

organizing machine data into events; and

creating a machine data web of linked events based on event data for the events.