SCALABLE INTERACTIVE DISPLAY OF DISTRIBUTED DATA

• US 20120239681A1
• Filed: 08/31/2011
• Published: 09/20/2012
• Est. Priority Date: 03/14/2011
• Status: Abandoned Application

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Sorkin, Stephen Phillip, ZHANG, Steve Yu
• Application Number:
  US13/223,167
• Publication Number:
  US 20120239681A1
• Time in Patent Office: Days
• US Class Current: 707/769
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24553 of query operations
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/90328 using search space presentation or visualization, e.g. category or range presentation and selection
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: H04L 41/0604 using filtering, e.g. reduction of information by using priority, element types, position or time
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method, system, and processor-readable storage medium are directed towards generating a report derived from data, such as event data, stored on a plurality of distributed nodes. In one embodiment the analysis is generated using a “divide and conquer” algorithm, such that each distributed node analyzes locally stored event data while an aggregating node combines these analysis results to generate the report. In one embodiment, each distributed node also transmits a list of event data references associated with the analysis result to the aggregating node. The aggregating node may then generate a global ordered list of data references based on the list of event data references received from each distributed node. Subsequently, in response to a user selection of a range of global event data, the report may dynamically retrieve event data from one or more distributed nodes for display according to the global order.

Claims

1. A method comprising:

receiving a query to search a plurality of event records that are stored across a plurality of computing devices;

submitting a sub-query derived from the received query to each of the plurality of computing devices, wherein each of the plurality of computing devices analyzes event records stored on that computing device, based on the sub-query, to generate one or more event references and to generate an analysis result;

receiving one or more event references from each of the plurality of computing devices, wherein each event reference includes a unique event record identifier and an event order value;

combining the one or more event references from each of the plurality of computing devices into a list of event references ordered by event order value;

providing the ordered list of event references in response to the query;

receiving a request for event records associated with a contiguous subset of the ordered list of event references;

requesting event records from one or more of the plurality of computing devices by sending each unique event record identifier included in the requested contiguous subset of the ordered list of event references to the corresponding computing device; and

providing the requested event records and the analysis result to a display device for display.

2. The method of claim 1, wherein each event reference of the ordered list of event references includes an identifier of the computing device that provided the event reference, and wherein retrieving event records associated with an event reference includes retrieving an event record from the computing device associated with the event reference.

3. The method of claim 1, wherein event records are analyzed by parsing a log file containing event records and returning event references for each event record satisfying the sub-query.

4. The method of claim 1, wherein sub-queries are submitted to computing devices that match a criteria included in the query.

5. The method of claim 1, wherein the event references are ordered based on an order indicated in the query.

6. The method of claim 1, wherein the query includes a first and a second ordering indication, wherein event references include a first order value and a second order value, wherein event references are ordered first by the first ordering indication, and wherein any event references that have the same first order value are ordered by the second order value.

7. The method of claim 1, wherein an event record includes raw event data, the unique event record identifier, and the event order value, and wherein the event references do not include raw event data.

8. An apparatus comprising:

a processor; and

a memory storing instructions that when executed by the processor cause actions to be performed, including;

receiving one or more event references from each of a plurality of computing devices, wherein each event reference includes an identifier of a source computing device, an index into an array of event records stored on the source computing device, and an event order value;

combining the one or more event references from each of the plurality of computing devices into a list of event references ordered by event order value;

receiving a request for event records associated with a contiguous range of the ordered list of event references;

retrieving event records from one or more of the plurality of computing devices by sending the index to the corresponding source computing device for each event reference in the requested contiguous range of the ordered list of event references; and

providing the requested event records to a display device for display.

9. The apparatus of claim 8, wherein the contiguous range of the ordered list of event references is selected from an interactive report displayed at the display device.

10. The apparatus of claim 8, wherein the one or more event references are received in response to a query received from the display device.

11. The apparatus of claim 8, wherein the plurality of computing devices are selected based on one or more of a form factor, an operating system, or a hardware component.

12. The apparatus of claim 8, wherein each computing device orders the one or more event references based on the event order value.

13. A non-transitory processor readable storage device storing instructions that cause a processor to perform actions, comprising:

receiving one or more event references from each of a plurality of computing devices, wherein each event reference includes an identifier of a source computing device, an index into an array of event records stored on the source computing device, and an event order value;

receiving, in response to a query request, statistical information from each of the plurality of computing devices, where the statistical information is generated by each of the plurality of computing devices performing an analysis of event records stored on that computing device;

combining the one or more event references from each of the plurality of computing devices into a list of event references, ordered by event order value;

combining the statistical information from each of the computing devices into an aggregated statistical information;

receiving a request for event records associated with a portion of the aggregated statistical information;

retrieving the requested event records from the plurality of computing devices by sending, for each event reference associated with the portion of the aggregated statistical information, the index into the array of event records stored on the source computing device to the corresponding source computing device; and

providing the requested event records to a display device for display.

14. The processor readable storage device of claim 13, wherein the statistical information includes an order statistic or an approximate order statistic.

15. The processor readable storage device of claim 13, wherein the statistical information includes a count of the number of event records matching a criteria included in a query received from the display device.

16. A system comprising:

a plurality of computing devices storing event records;

a first computing device configured to perform actions comprising;

receiving one or more event references from each of the plurality of computing devices, wherein each event reference includes a unique event record identifier and an event order value;

combining the one or more event references from each of the plurality of computing devices into a list of event references ordered by event order value;

receiving a request for event records associated with a contiguous range of ordered event references;

retrieving event records from one or more of the plurality of computing devices by sending each unique event record identifier included in the requested contiguous range of ordered event references to the corresponding computing device; and

providing the requested event records for display.

17. The system of claim 16, wherein each event reference of the ordered list of event references includes an identifier of the computing device that provided the event reference, and wherein retrieving event records associated with an event reference includes retrieving an event record from the computing device associated with the event reference.

18. The system of claim 16, the actions further comprising:

parsing a log file containing event records to generate a statistical analysis; and

providing the statistical analysis event references for each event record that satisfy a query.

19. The system of claim 16, wherein an event record includes data collected about a computing system and is stored on the same computing system.

20. The system of claim 16, wherein the ordered list of event references is ordered by a first order value and a second order value, wherein event references are ordered first by the first order value, and wherein any event references that have the same first order value are ordered by the second order value.