Collection query driven generation of summarization information for raw machine data

• US 10,387,396 B2
• Filed: 09/15/2017
• Issued: 08/20/2019
• Est. Priority Date: 01/31/2013
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Marquardt, David Ryan, Sorkin, Stephen Phillip, Zhang, Steve Yu
• Primary Examiner(s): Pham, Tuan A
• Application Number:
  US15/705,875
• Publication Number:
  US 20180004785A1
• Time in Patent Office: 704 Days
• Field of Search: None
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24539 using cached or materialised query results
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Embodiments are directed are towards the transparent summarization of events. Queries directed towards summarizing and reporting on event records may be received at a search head. Search heads may be associated with one more indexers containing event records. The search head may forward the query to the indexers the can resolve the query for concurrent execution. If a query is a collection query, indexers may generate summarization information based on event records located on the indexers. Event record fields included in the summarization information may be determined based on terms included in the collection query. If a query is a stats query, each indexer may generate a partial result set from previously generated summarization information, returning the partial result sets to the search head. Collection queries may be saved and scheduled to run and periodically update the summarization information.

Claims

1. A method comprising:

providing a field searchable data store comprising a plurality of partitions of field searchable, time stamped event records, each event record comprising a time-stamped portion of raw machine data;

receiving a collection query that references a field name, wherein the field name corresponds to at least one field value, and wherein the collection query comprises commands to generate summarization information for one or more field names included therein;

responsive to the collection query, generating a respective summarization table for each partition of field searchable, time stamped event records by;

forwarding the collection query to an indexer, wherein the indexer comprises one or more partitions of field searchable, time stamped event records of the plurality of partitions;

determining partitions of field searchable, time stamped event records responsive to the collection query;

determining an extraction rule associated with the field name, wherein the extraction rule comprises instructions applied to identify and extract a field value associated with the field name;

extracting the field value corresponding to the field name from one or more event records in responsive partitions using the extraction rule; and

populating the respective summarization table responsive to each extracted field value, wherein each entry comprises the field name, the corresponding field value and a posting value that identifies a location in a corresponding partition where an associated event record is stored

receiving a first incoming search query from a search head;

generating a partial result to the first incoming search query using summarization tables generated corresponding to each partition of field searchable, time stamped event records; and

generating a result set responsive to the first incoming search query, wherein the result set is generated using the partial result, wherein the search head is operable to combine partial result sets returned from each partition of field searchable, time stamped event records to generate the result set.

2. The method of claim 1, wherein the forwarding further comprises:

forwarding the collection query to a plurality of indexers concurrently, wherein each indexer comprises one or more partitions of field searchable, time stamped event records of the plurality of partitions relevant to the collection query.

3. The method of claim 1, wherein the responsive partitions are associated with a time-frame specified in the collection query.

4. The method of claim 1, wherein each of the plurality of partitions of field searchable, time stamped event records are stored at a different one of a plurality of distributed locations in a distributed indexed data store.

5. The method of claim 1, wherein each of the plurality of partitions of field searchable, time stamped event records are stored at a different one of a plurality of distributed locations in a distributed indexed data store, wherein each of the plurality of distributed locations is included in a different indexer that manages a partition of the plurality of partitions of field searchable, time stamped event records.

6. The method of claim 1, further comprising:

storing the respective summarization table at an associated indexer that manages the responsive partitions.

7. The method of claim 1, wherein the field searchable data store comprises a distributed indexed data store stored in a distributed manner among two or more servers.

8. The method of claim 1, wherein the respective summarization table generated for each partition of field searchable, time stamped event records is associated with a search head where the collection query originated.

9. The method of claim 1, further comprising:

responsive to a second incoming search query that references the field name, retrieving the respective summarization table generated for each partition of field searchable, time stamped event records and using partial summarization information in each respective summarization table to respond to the second incoming search query.

10. The method of claim 1, further comprising:

receiving a second incoming search query;

generating a partial result for the second incoming search query using corresponding summarization tables generated for each partition of field searchable, time stamped event records, wherein the partial result is generated without evaluating each event record in associated event records; and

generating a result set responsive to the second incoming search query, wherein the result set is generated using the partial result.

11. The method of claim 1, further comprising:

receiving a second incoming search query;

generating a partial result for the second incoming search query using summarization tables corresponding to each partition of field searchable, time stamped event records, wherein the partial result is generated without evaluating each event record in associated event records;

evaluating each partition of field searchable, time stamped event records to include fields omitted from an associated summarization table; and

generating a result set responsive to the second incoming search query, wherein the result set is generated using the partial result and the omitted fields.

12. The method of claim 1, wherein the collection query is provided to a search head from a user via a user-interface.

13. The method of claim 1, wherein the collection query is received through an application programming interface (API).

14. The method of claim 1, wherein the collection query is a saved query scheduled to execute periodically.

15. The method of claim 1, wherein the collection query is a saved query scheduled to execute on the occurrence of a condition.

16. A non-transitory computer-readable medium storing computer-executable instructions which, when executed by a processor, cause the processor to perform operations comprising:

providing a field searchable data store comprising a plurality of partitions of field searchable, time stamped event records, each event record comprising a time-stamped portion of raw machine data;

receiving a collection query that references a field name, wherein the field name corresponds to at least one field value, and wherein the collection query comprises commands to generate summarization information for one or more field names included therein;

responsive to the collection query, generating a respective summarization table for each partition of field searchable, time stamped event records by;

forwarding the collection query to an indexer, wherein the indexer comprises one or more partitions of field searchable, time stamped event records of the plurality of partitions;

determining partitions of field searchable, time stamped event records responsive to the collection query;

determining an extraction rule associated with the field name, wherein the extraction rule comprises instructions applied to identify and extract a field value associated with the field name;

extracting the field value corresponding to the field name from one or more event records in responsive partitions using the extraction rule; and

populating the respective summarization table responsive to each extracted field value, wherein each entry comprises the field name, the corresponding field value and a posting value that identifies a location in a corresponding partition where an associated event record is stored;

receiving a first incoming search query from a search head;

generating a partial result to the first incoming search query using summarization tables generated corresponding to each partition of field searchable, time stamped event records; and

generating a result set responsive to the first incoming search query, wherein the result set is generated using the partial result, wherein the search head is operable to combine partial result sets returned from each partition of field searchable, time stamped event records to generate the result set.

17. The non-transitory computer-readable medium of claim 16, wherein the forwarding operation further comprises:

forwarding the collection query to a plurality of indexers concurrently, wherein each indexer comprises one or more partitions of field searchable, time stamped event records of the plurality of partitions relevant to the collection query.

18. The non-transitory computer-readable medium of claim 16, wherein the responsive partitions are associated with a time-frame specified in the collection query.

19. The non-transitory computer-readable medium of claim 16, wherein each of the plurality of partitions of field searchable, time stamped event records are stored at a different one of a plurality of distributed locations in a distributed indexed data store.

20. The non-transitory computer-readable medium of claim 16, wherein each of the plurality of partitions of field searchable, time stamped event records are stored at a different one of a plurality of distributed locations in a distributed indexed data store, wherein each of the plurality of distributed locations is included in a different indexer that manages a partition of the plurality of partitions of field searchable, time stamped event records.

21. The non-transitory computer-readable medium of claim 16, further comprising:

storing the summarization table at an associated indexer that manages the responsive partitions.

22. The non-transitory computer-readable medium of claim 16, wherein the field searchable data store comprises a distributed indexed data store stored in a distributed manner among two or more servers.

23. The non-transitory computer-readable medium of claim 16, wherein the respective summarization table generated for each partition of field searchable, time stamped event records is associated with a search head where the collection query originated.

24. The non-transitory computer-readable medium of claim 16, wherein the operations further comprise:

responsive to a second incoming search query that references the field name, retrieving the respective summarization table generated for each partition of field searchable, time stamped event records and using partial summarization information in each summarization table to respond to the second incoming search query.

25. A system comprising:

at least one memory storing computer-executable instructions; and

at least one processor, wherein the at least one processor is configured to access the at least one memory and to execute the computer-executable instructions to;

provide a field searchable data store comprising a plurality of partitions of field searchable, time stamped event records, each event record comprising a time-stamped portion of raw machine data;

receive a collection query that references a field name, wherein the field name corresponds to at least one field value, and wherein the collection query comprises commands to generate summarization information for one or more field names included therein;

responsive to the collection query, generate a respective summarization table for each partition of field searchable, time stamped event records by;

forwarding the collection query to an indexer, wherein the indexer comprises one or more partitions of field searchable, time stamped event records of the plurality of partitions;

determining partitions of field searchable, time stamped event records responsive to the collection query;

determining an extraction rule associated with the field name, wherein the extraction rule comprises instructions applied to identify and extract a field value associated with the field name;

extracting the field value corresponding to the field name from one or more event records in responsive partitions using the extraction rule; and

populating the respective summarization table responsive to each extracted field value, wherein each entry comprises the field name, the corresponding field value and a posting value that identifies a location in a corresponding partition where an associated event record is stored;

receiving a first incoming search query from a search head;

generating a partial result to the first incoming search query using summarization tables generated corresponding to each partition of field searchable, time stamped event records; and

generating a result set responsive to the first incoming search query, wherein the result set is generated using the partial result, wherein the search head is operable to combine partial result sets returned from each partition of field searchable, time stamped event records to generate the result set.

26. The system of claim 25, wherein the processor is further configured to access at least one memory and to execute the computer-executable instructions to:

responsive to a second incoming search query that references the field name, retrieve the respective summarization table generated for each partition of field searchable, time stamped event records and using partial summarization information in each summarization table to respond to the second incoming search query.

27. The system of claim 25, wherein the processor is further configure to access the least one memory and to execute the computer-executable instructions to:

receive a second incoming search query;

generate a partial result for the second incoming search query using summarization tables corresponding to each partition of field searchable, time stamped event records, wherein the partial result is generated without evaluating each event record in associated event records; and

generate a result set responsive to the second incoming search query, wherein the result set is generated using the partial result.