Distributed high performance analytics store

• US 8,682,925 B1
• Filed: 01/31/2013
• Issued: 03/25/2014
• Est. Priority Date: 01/31/2013
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Marquardt, David Ryan, Sorkin, Stephen Phillip, Zhang, Steve Yu
• Primary Examiner(s): Jalil, Neveen Abel
• Assistant Examiner(s): Tamaru, Michael K
• Application Number:
  US13/756,147
• Time in Patent Office: 418 Days
• Field of Search: None
• US Class Current: 707/770
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24539 using cached or materialised query results
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Embodiments are directed are towards the transparent summarization of events. Queries directed towards summarizing and reporting on event records may be received at a search head. Search heads may be associated with one more indexers containing event records. The search head may forward the query to the indexers the can resolve the query for concurrent execution. If a query is a collection query, indexers may generate summarization information based on event records located on the indexers. Event record fields included in the summarization information may be determined based on terms included in the collection query. If a query is a stats query, each indexer may generate a partial result set from previously generated summarization information, returning the partial result sets to the search head. Collection queries may be saved and scheduled to run and periodically update the summarization information.

Claims

1. A computer-implemented method for generating a result set among a plurality of distributed locations, the method comprising:

receiving, at a computing device, raw data;

segmenting the raw data into a set of time-stamped event records;

dividing the set of time-stamped event records into two or more partitions of event records;

indexing and storing each time-stamped event record of each of the two or more partitions of event records, wherein each of the two or more partitions of event records are stored at a different one of a plurality of distributed locations in a distributed indexed data store;

for each partition of event records, generating a summarization table that;

identifies one or more field-value combinations, wherein a field-value combination includes a field and a value that appears in one or more of the event records of that partition for that field, and wherein a data model or a command is used to identify one or more fields for inclusion in the summarization table; and

for each field-value combination, identifies a set of one or more posting values of event records of that partition that have the value for the field, wherein a posting value provides a way to retrieve the event record to which it corresponds from the distributed indexed data store;

receiving a query;

generating one or more partial results for the query, wherein the one or more partial results are generated using the summarization table for a partition of event records, and wherein the one or more partial results are generated without evaluating each individual event record in the partition of event records; and

generating a result set responsive to the query, wherein the result set is generated using the one or more partial results.

2. The method of claim 1, wherein each of the plurality of distributed locations is included in a different indexer that manages a partition of event records.

3. The method of claim 1, wherein each of the plurality of distributed locations is included in a different indexer that manages two or more partitions of event records.

4. The method of claim 1, further comprising:

storing the summarization table for a partition of event records at an indexer that manages the partition of event records.

5. The method of claim 1, further comprising:

determining for the partition of event records whether the one or more partial results for the query can be generated using the summarization table for the partition and determining for the partition whether the one or more partial results can be generated without evaluating each individual event record in the partition.

6. The method of claim 1, further comprising:

determining for the partition of events whether the one or more partial results can be generated without evaluating at least one individual event record in the partition

based on determining that the one or more partial results for the query cannot be generated without evaluating at least one individual event record in the partition, evaluating one or more individual event records in the partition to generate the one or more partial results.

7. The method of claim 1, further comprising:

determining that the partition of event records includes one or more event records that have not been processed for inclusion in the summarization table for the partition; and

producing the one or more partial results for the query by evaluating the one or more event records that have not been processed for inclusion in the summarization table for the partition.

8. The method of claim 1, wherein the one or more partial results are generated from one or more event records in at least one of the two or more partitions of event records.

9. A computer-implemented system for generating a result set among a plurality of distributed locations said system comprising:

one or more processors;

one or more non-transitory computer-readable storage mediums containing instructions configured to cause the one or more processors to perform operations including;

receiving raw data;

segmenting the raw data into a set of time-stamped event records;

dividing the set of time-stamped event records events into two or more partitions of event records;

indexing and storing each time-stamped event record of each of the two or more partitions of event records, wherein each of the two or more partitions of event records are stored at a different one of a plurality of distributed locations in a distributed indexed data store;

for each partition of event records, generating a summarization table that;

identifies one or more field-value combinations, wherein a field-value combination includes a field and a value that appears in one or more of the event records of that partition for that field, and wherein a data model or a command is used to identify one or more fields for inclusion in the summarization table; and

for each field-value combination, identifies a set of one or more posting values of event records of that partition that have the value for the field, wherein a posting value provides a way to retrieve the event record to which it corresponds from the distributed indexed data store;

receiving a query;

generating one or more partial results for the query, wherein the one or more partial results are generated using the summarization table for a partition of event records, and wherein the one or more partial results are generated without evaluating each individual event record in the partition of event records; and

generating a result set responsive to the query, wherein the result set is generated using the one or more partial results.

10. The system of claim 9, wherein each of the plurality of distributed locations is included in a different indexer that manages a partition of event records.

11. The system of claim 9, wherein each of the plurality of distributed locations is included in a different indexer that manages two or more partitions of event records.

12. The system of claim 9, further comprising instructions configured to cause the one or more processors to perform operations including:

storing the summarization table for a partition of event records at an indexer that manages the partition of event records.

13. The system of claim 9, further comprising instructions configured to cause the one or more processors to perform operations including:

determining for the partition of event records whether the one or more partial results for the query can be generated using the summarization table for the partition and determining for the partition whether the one or more partial results can be generated without evaluating each individual event record in the partition.

14. The system of claim 9, further comprising instructions configured to cause the one or more processors to perform operations including:

determining for the partition of events whether the one or more partial results can be generated without evaluating at least one individual event record in the partition

based on determining that the one or more partial results for the query cannot be generated without evaluating at least one individual event record in the partition, evaluating one or more individual event records in the partition to generate the one or more partial results.

15. The system of claim 9, further comprising instructions configured to cause the one or more processors to perform operations including:

determining that the partition of event records includes one or more event records that have not been processed for inclusion in the summarization table for the partition; and

producing the one or more partial results for the query by evaluating the one or more event records that have not been processed for inclusion in the summarization table for the partition.

16. The system of claim 9, wherein the one or more partial results are generated from one or more event records in at least one of the two or more partitions of event records.

17. A computer-program product for generating a result set among a plurality of distributed locations, tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause a data processing apparatus to:

receive raw data;

segment the raw data into a set of time-stamped event records;

divide the set of time-stamped event records into two or more partitions of event records;

index and store each time-stamped event record of each of the two or more partitions of event records wherein each of the two or more partitions of event records are stored at a different one of a plurality of distributed locations in a distributed indexed data store;

for each partition of event records, generate a summarization table that;

identifies one or more field-value combinations, wherein a field-value combination includes a field and a value that appears in one or more of the event records of that partition for that field, and wherein a data model or a command is used to identify one or more fields for inclusion in the summarization table; and

for each field-value combination, identifies a set of one or more posting values of event records of that partition that have the value for the field, wherein a posting value provides a way to retrieve the event record to which it corresponds from the distributed indexed data store;

receive a query;

generate one or more partial results for the query, wherein the one or more partial results are generated using the summarization table for a partition of event records, and wherein the one or more partial results are generated without evaluating each individual event record in the partition of event records; and

generate a result set responsive to the query, wherein the result set is generated using the one or more partial results.

18. The computer-program product of claim 17, wherein each of the plurality of distributed locations is included in a different indexer that manages a partition of event records.

19. The computer-program product of claim 17, wherein each of the plurality of distributed locations is included in a different indexer that manages two or more partitions of event records.

20. The computer-program product of claim 17, further comprising instructions configured to cause a data processing apparatus to:

store the summarization table for a partition of event records at an indexer that manages the partition of event records.

21. The computer-program product of claim 17, further comprising instructions configured to cause a data processing apparatus to:

determine for the partition of event records whether the one or more partial results for the query can be generated using the summarization table for the partition and determine for the partition whether the one or more partial results can be generated without evaluating each individual event record in the partition.

22. The computer-program product of claim 17, further comprising instructions configured to cause a data processing apparatus to:

determine for the partition of events whether the one or more partial results can be generated without evaluating at least one individual event record in the partition

based on determining that the one or more partial results for the query cannot be generated without evaluating at last one individual event record in the partition, evaluate one or more individual event records in the partition to generate the one or more partial results.

23. The computer-program product of claim 17, wherein the one or more partial results are generated from one or more event records in at least one of the two or more partitions of event records.