Query handling using summarization tables

• US 10,685,001 B2
• Filed: 04/30/2018
• Issued: 06/16/2020
• Est. Priority Date: 01/31/2013
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Marquardt, David Ryan, Sorkin, Stephen Phillip, Zhang, Steve Yu
• Primary Examiner(s): Dang, Thanh-Ha
• Application Number:
  US15/967,400
• Publication Number:
  US 20180246918A1
• Time in Patent Office: 778 Days
• Field of Search: 707769
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24539 using cached or materialised query results
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Embodiments are directed are towards the transparent summarization of events. Queries directed towards summarizing and reporting on event records may be received at a search head. Search heads may be associated with one more indexers containing event records. The search head may forward the query to the indexers the can resolve the query for concurrent execution. If a query is a collection query, indexers may generate summarization information based on event records located on the indexers. Event record fields included in the summarization information may be determined based on terms included in the collection query. If a query is a stats query, each indexer may generate a partial result set from previously generated summarization information, returning the partial result sets to the search head. Collection queries may be saved and scheduled to run and periodically update the summarization information.

Claims

1. A method, comprising:

creating a set of field searchable, time stamped event records from raw data stored in at least one datastore, wherein each field searchable, time stamped event record in the set of field searchable, time stamped event records comprises a portion of the raw data and is associated with a time stamp derived from the raw data;

generating a summarization table for a set of field names in the set of field searchable, time stamped event records that identifies one or more field values associated with the set of field names and further generating, for each field value, one or more posting values to field searchable, time stamped event records in the at least one data store having the field value, wherein a field value comprises a value that appears in connection with an associated field name in one or more field searchable, time stamped event records in the set of field searchable, time stamped event records, and wherein each posting value of the one or more posting values references a location of a corresponding field searchable, time stamped event record in the at least one data store;

storing the summarization table;

receiving a search query that includes search criteria for evaluating field values for one or more field names;

using the search criteria to evaluate the field values for the one or more field names in the summarization table to generate a query result;

causing display of information based on the query result.

2. The method of claim 1, further comprising:

executing an action based on the query result.

3. The method of claim 1, further comprising:

based on using the search criteria to evaluate the field values for the one or more field names in the summarization table, retrieving event records identified in the summarization table for further processing to generate the query result.

4. The method of claim 1, wherein the wherein the event records identified in the summarization table are retrieved using the one or more posting values.

5. The method of claim 1, further comprising:

storing the set of field searchable, time stamped event records in an indexed datastore.

6. The method of claim 1, further comprising:

storing the set of field searchable, time stamped event records in an indexed datastore, wherein the indexed datastore comprises a distributed indexed datastore.

7. The method of claim 1, further comprising:

storing the set of field searchable, time stamped event records in an indexed datastore, wherein the indexed datastore is stored in a distributed manner among two or more servers.

8. The method of claim 1, further comprising:

receiving a command identifying field names to include in the summarization table for the set of field searchable, time stamped event records.

9. The method of claim 1, wherein the generating is responsive to a collection query that references the set of field names to include in the summarization table.

10. The method of claim 1, further comprising:

receiving a collection query identifying field names to include in the summarization table for the set of field searchable, time stamped event records; and

scheduling the collection query to run periodically to update summarization information corresponding to the collection query.

11. The method of claim 1, further comprising:

in response to determining that summarization information in the summarization table is incomplete, generating summarization information for event records responsive to the search query in real time to add to the query result.

12. The method of claim 1, wherein the raw data comprises machine data.

13. The method of claim 1, wherein the raw data comprises log data.

14. The method of claim 1, wherein the raw data comprises unstructured data.

15. The method of claim 1, wherein the summarization table comprises two or more table portions, and wherein the two or more table portions are stored in a distributed manner.

16. The method of claim 1, wherein the summarization table comprises two or more table portions, wherein each of the two or more table portions is stored in a distributed manner such that each portion is proximate to a subset of event records to which each respective table portion it-pertains.

17. The method of claim 1, wherein the set of field searchable, time stamped event records are distributed across one or more partitions of field searchable, time stamped event records, wherein the summarization table comprises one or more summary partitions, and wherein each of the one or more summary partitions contains summary information associated with a respective partition of field searchable, time stamped event records.

18. A non-transitory computer-readable medium storing computer-executable instructions which, when executed by a processor, cause the processor to perform operations comprising:

creating a set of field searchable, time stamped event records from raw data stored in at least one datastore, wherein each field searchable, time stamped event record in the set of field searchable, time stamped event records comprises a portion of the raw data and is associated with a time stamp derived from the raw data;

generating a summarization table for a set of field names in the set of field searchable, time stamped event records that identifies one or more field values associated with the set of field names and further generating, for each field value, one or more posting values to field searchable, time stamped event records in the at least one data store having the field value, wherein a field value comprises a value that appears in connection with an associated field name in one or more field searchable, time stamped event records in the set of field searchable, time stamped event records, and wherein each posting value of the one or more posting values references a location of a corresponding field searchable, time stamped event record in the at least one data store;

storing the summarization table;

receiving a search query that includes search criteria for evaluating field values for one or more field names;

using the search criteria to evaluate the field values for the one or more field names in the summarization table to generate a query result;

causing display of information based on the query result.

19. The non-transitory computer-readable medium of claim 18, further comprising:

based on using the search criteria to evaluate the field values for the one or more field names in the summarization table, retrieving event records identified in the summarization table for further processing to generate the query result.

20. The non-transitory computer-readable medium of claim 18, wherein the wherein the event records identified in the summarization table are retrieved using the one or more posting values.

21. The non-transitory computer-readable medium of claim 18, further comprising:

storing the set of field searchable, time stamped event records in an indexed datastore.

22. The non-transitory computer-readable medium of claim 18, further comprising:

storing the set of field searchable, time stamped event records in an indexed datastore, wherein the indexed datastore comprises a distributed indexed datastore.

23. The non-transitory computer-readable medium of claim 18, further comprising:

storing the set of field searchable, time stamped event records in an indexed datastore, wherein the indexed datastore is stored in a distributed manner among two or more servers.

24. The non-transitory computer-readable medium of claim 18, further comprising:

receiving a command identifying field names to include in the summarization table for the set of field searchable, time stamped event records.

25. The non-transitory computer-readable medium of claim 18, wherein the generating is responsive to a collection query that references the set of field names to include in the summarization table.

26. The non-transitory computer-readable medium of claim 18, further comprising:

receiving a collection query identifying field names to include in the summarization table for the set of field searchable, time stamped event records; and

scheduling the collection query to run periodically to update summarization information corresponding to the collection query.

27. The non-transitory computer-readable medium of claim 18, further comprising:

in response to determining that summarization information in the summarization table is incomplete, generating summarization information for event records responsive to the search query in real time to add to the query result.

28. The non-transitory computer-readable medium of claim 18, wherein the summarization table comprises two or more table portions, and wherein the two or more table portions are stored in a distributed manner.

29. The non-transitory computer-readable medium of claim 18, wherein the summarization table comprises two or more table portions, wherein each of the two or more table portions is stored in a distributed manner such that each portion is proximate to a subset of event records to which each respective table portion pertains.

30. The non-transitory computer-readable medium of claim 18, wherein the set of field searchable, time stamped event records are distributed across one or more partitions of field searchable, time stamped event records, wherein the summarization table comprises one or more summary partitions, and wherein each of the one or more summary partitions contains summary information associated with a respective partition of field searchable, time stamped event records.

31. A system comprising:

at least one memory storing computer-executable instructions; and

at least one processor, wherein the at least one processor is configured to access the at least one memory and to execute the computer-executable instructions to;

create a set of field searchable, time stamped event records from raw data stored in at least one datastore, wherein each field searchable, time stamped event record in the set of field searchable, time stamped event records comprises a portion of the raw data and is associated with a time stamp derived from the raw data;

generate a summarization table for a set of field names in the set of field searchable, time stamped event records that identifies one or more field values associated with the set of field names and further generate, for each field value, one or more posting values to field searchable, time stamped event records in the at least one data store having the field value, wherein a field value comprises a value that appears in connection with an associated field name in one or more field searchable, time stamped event records in the set of field searchable, time stamped event records, and wherein each posting value of the one or more posting values references a location of a corresponding field searchable, time stamped event record in the at least one data store;

store the summarization table;

receive a search query that includes search criteria for evaluating field values for one or more field names;

use the search criteria to evaluate the field values for the one or more field names in the summarization table to generate a query result;

cause display of information based on the query result.