Supplementing a high performance analytics store with evaluation of individual events to respond to an event query

• US 9,128,985 B2
• Filed: 01/31/2014
• Issued: 09/08/2015
• Est. Priority Date: 01/31/2013
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Marquardt, David Ryan, Sorkin, Stephen Phillip, Zhang, Steve Yu
• Primary Examiner(s): Chbouki, Tarek
• Assistant Examiner(s): Tamaru, Michael K
• Application Number:
  US14/170,159
• Publication Number:
  US 20140214888A1
• Time in Patent Office: 585 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24539 using cached or materialised query results
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Embodiments are directed are towards the transparent summarization of events. Queries directed towards summarizing and reporting on event records may be received at a search head. Search heads may be associated with one more indexers containing event records. The search head may forward the query to the indexers the can resolve the query for concurrent execution. If a query is a collection query, indexers may generate summarization information based on event records located on the indexers. Event record fields included in the summarization information may be determined based on terms included in the collection query. If a query is a stats query, each indexer may generate a partial result set from previously generated summarization information, returning the partial result sets to the search head. Collection queries may be saved and scheduled to run and periodically update the summarization information.

Claims

1. A computer implemented method, comprising:

receiving raw data at a computing device;

parsing the raw data into event records by determining event boundaries in the raw data, wherein each of the event records includes a portion of the raw data and is associated with a time derived from the raw data;

storing the event records in an indexed data store;

generating a summarization table that;

identifies one or more field values, wherein a field value comprises a value that appears in an associated field in one or more of the event records in the indexed data store; and

for each field value, identifies a set of one or more event records in the indexed data store that contain the field value for the associated field;

receiving a query that includes search criteria for evaluating field values for one or more fields;

using the search criteria to evaluate field values for one or more fields in the summarization table to generate a preliminary result set;

determining that the query cannot be answered fully by the summarization table by determining that the indexed data store includes event records that have not been processed for inclusion in the summarization table; and

based on determining that the indexed data store includes event records that have not been processed for inclusion in the summarization table;

using the search criteria to identify supplemental event records in the indexed data store that satisfy the search criteria and that have not been processed for inclusion in the summarization table;

generating a query result using the preliminary result set from the summarization table and the supplemental event records; and

causing display of the query result or transmitting the query result to a second computing device for further processing and output.

2. The method of claim 1, wherein the indexed data store comprises a distributed indexed data store.

3. The method of claim 1, wherein the indexed data store is stored in a distributed manner among two or more servers.

4. The method of claim 1, further comprising receiving a command identifying fields to include in the summarization table.

5. The method of claim 1, further comprising:

storing a data model that identifies fields in one or more event records; and

including fields in the stored data model in the summarization table.

6. The method of claim 1, wherein the raw data includes machine data.

7. The method of claim 1, wherein the raw data includes log data.

8. The method of claim 1, wherein the raw data includes unstructured data.

9. The method of claim 1, wherein the summarization table includes two or more table portions, and wherein the two or more table portions are stored in a distributed manner.

10. The method of claim 1, wherein the summarization table includes two or more table portions, and wherein each of the two or more table portions is stored in a distributed manner proximate to a subset of the event records to which it pertains.

11. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to perform:

receiving raw data at a computing device;

parsing the raw data into event records by determining event boundaries in the raw data, wherein each of the event records includes a portion of the raw data and is associated with a time derived from the raw data;

storing the event records in an indexed data store;

generating a summarization table that;

identifies one or more field values, wherein a field value comprises a value that appears in an associated field in one or more of the event records in the indexed data store; and

for each field value, identifies a set of one or more event records in the indexed data store that contain the field value for the associated field;

receiving a query that includes search criteria for evaluating field values for one or more fields;

using the search criteria to evaluate field values for one or more fields in the summarization table to generate a preliminary result set;

determining that the query cannot be answered fully by the summarization table by determining that the indexed data store includes event records that have not been processed for inclusion in the summarization table; and

based on determining that the indexed data store includes event records that have not been processed for inclusion in the summarization table;

using the search criteria to identify supplemental event records in the indexed data store that satisfy the search criteria and that have not been processed for inclusion in the summarization table;

generating a query result using the preliminary result set from the summarization table and the supplemental event records; and

causing display of the query result or transmitting the query result to a second computing device for further processing and output.

12. The computer-readable medium of claim 11, wherein the indexed data store comprises a distributed indexed data store.

13. The computer-readable medium of claim 11, wherein the indexed data store is stored in a distributed manner among two or more servers.

14. The computer-readable medium of claim 11, further comprising receiving a command identifying fields to include in the summarization table.

15. The computer-readable medium of claim 11, further comprising:

storing a data model that identifies fields in one or more event records; and

including fields in the stored data model in the summarization table.

16. The computer-readable medium of claim 11, wherein the raw data includes machine data.

17. The computer-readable medium of claim 11, wherein the raw data includes log data.

18. The computer-readable medium of claim 11, wherein the raw data includes unstructured data.

19. The computer-readable medium of claim 11, wherein the summarization table includes two or more table portions, and wherein the two or more table portions are stored in a distributed manner.

20. The computer-readable medium of claim 11, wherein the summarization table includes two or more table portions, and wherein each of the two or more table portions is stored in a distributed manner proximate to a subset of the event records to which it pertains.

21. An apparatus, comprising:

an indexed data store;

a subsystem, implemented at least partially in hardware, that receives raw data at a computing device;

a subsystem, implemented at least partially in hardware, that parses the raw data into event records by determining event boundaries in the raw data, wherein each of the event records includes a portion of the raw data and is associated with a time derived from the raw data;

a subsystem, implemented at least partially in hardware, that stores the event records in the indexed data store;

a subsystem, implemented at least partially in hardware, that generates a summarization table that;

identifies one or more field values, wherein a field value comprises a value that appears in an associated field in one or more of the event records in the indexed data store; and

for each field value, identifies a set of one or more event records in the indexed data store that contain the field value for the associated field;

a subsystem, implemented at least partially in hardware, that receives a query that includes search criteria for evaluating field values for one or more fields;

a subsystem, implemented at least partially in hardware that uses the search criteria to evaluate field values for one or more fields in the summarization table to generate a preliminary result set;

a query determination subsystem, implemented at least partially in hardware, that determines that the query cannot be answered fully by the summarization table by determining that the indexed data store includes event records that have not been processed for inclusion in the summarization table; and

a subsystem, implemented at least partially in hardware that, based on determining that the indexed data store includes event records that have not been processed for inclusion in the summarization table;

using the search criteria to identify supplemental event records in the indexed data store that satisfy the search criteria and that have not been processed for inclusion in the summarization table;

generating a query result using the preliminary result set from the summarization table and the supplemental event records; and

causing display of the query result or transmitting the query result to a second computing device for further processing and output.

22. The apparatus of claim 21, wherein the indexed data store comprises a distributed indexed data store.

23. The apparatus of claim 21, wherein the indexed data store is stored in a distributed manner among two or more servers.

24. The apparatus of claim 21, further comprising receiving a command identifying fields to include in the summarization table.

25. The apparatus of claim 21, further comprising:

a subsystem, implemented at least partially in hardware, that stores a data model that identifies fields in one or more event records; and

a subsystem, implemented at least partially in hardware, that includes fields in the stored data model in the summarization table.

26. The apparatus of claim 21, wherein the raw data includes machine data.

27. The apparatus of claim 21, wherein the raw data includes log data.

28. The apparatus of claim 21, wherein the raw data includes unstructured data.

29. The apparatus of claim 21, wherein the summarization table includes two or more table portions, and wherein the two or more table portions are stored in a distributed manner.

30. The apparatus of claim 21, wherein the summarization table includes two or more table portions, and wherein each of the two or more table portions is stored in a distributed manner proximate to a subset of the event records to which it pertains.