GENERATING AND STORING SUMMARIZATION TABLES FOR SETS OF SEARCHABLE EVENTS

• US 20180246918A1
• Filed: 04/30/2018
• Published: 08/30/2018
• Est. Priority Date: 01/31/2013
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Marquardt, David Ryan, Sorkin, Stephen Phillip, Zhang, Steve Yu
• Granted Patent: US 10,685,001 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24539 using cached or materialised query results
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Embodiments are directed are towards the transparent summarization of events. Queries directed towards summarizing and reporting on event records may be received at a search head. Search heads may be associated with one more indexers containing event records. The search head may forward the query to the indexers the can resolve the query for concurrent execution. If a query is a collection query, indexers may generate summarization information based on event records located on the indexers. Event record fields included in the summarization information may be determined based on terms included in the collection query. If a query is a stats query, each indexer may generate a partial result set from previously generated summarization information, returning the partial result sets to the search head. Collection queries may be saved and scheduled to run and periodically update the summarization information.

Claims

1. A method, comprising:

creating two or more sets of searchable, time stamped event records from raw data stored in at least one data store, wherein each searchable, time stamped event record in the two or more sets of searchable, time stamped event records includes a portion of the raw data and is associated with a time stamp derived from the raw data, wherein the raw data reflects activity in an information technology environment;

generating a summarization table for each set of searchable, time stamped event records in the two or more sets of searchable, time stamped event records that;

identifies one or more field values, wherein a field value comprises a value that appears in an associated field in one or more searchable, time stamped event records in the set of searchable, time stamped event records; and

for each field value, identifies the one or more searchable, time stamped event records in the set of searchable, time stamped event records that contain the field value for the associated field;

storing the summarization table for each set of searchable, time stamped event records among the two or more sets of time stamped searchable event records;

selecting a stored summarization table based on a received query that includes search criteria for evaluating field values for one or more fields;

using the search criteria to evaluate field values for one or more fields in the selected summarization table to generate a query result; and

wherein the query result reflects an aspect of activity in the information technology environment.

2. The method of claim 1, further comprising:

causing display of information based on the query result.

3. The method of claim 1, further comprising:

causing display of the query result.

4. The method of claim 1, further comprising:

executing an action based on the query result.

5. The method of claim 1, further comprising:

based on using the search criteria to evaluate field values for one or more fields in the selected summarization table, retrieving event records identified in the summarization table for further processing to generate the query result.

6. The method of claim 1, wherein the query result is generated from the summarization table with retrieving event records identified in the summarization table.

7. The method of claim 1, further comprising:

storing the two or more sets of searchable, time stamped event records in an indexed data store.

8. The method of claim 1, further comprising:

storing the two or more sets of searchable, time stamped event records in an indexed data store;

wherein the indexed data store comprises a distributed indexed data store.

9. The method of claim 1, further comprising:

storing the two or more sets of searchable, time stamped event records in an indexed data store;

wherein the indexed data store is stored in a distributed manner among two or more servers.

10. The method of claim 1, further comprising:

receiving a command identifying fields to include in the summarization table for each set of searchable, time stamped event records in the two or more sets of searchable, time stamped event records.

11. The method of claim 1, wherein the raw data includes machine data.

12. The method of claim 1, wherein the raw data includes log data.

13. The method of claim 1, wherein the raw data includes unstructured data.

14. The method of claim 1, wherein the summarization table for each set of searchable, time stamped event records in the one or more sets of searchable, time stamped event records includes two or more table portions, and wherein the two or more table portions are stored in a distributed manner.

15. The method of claim 1, wherein the summarization table for each set of searchable, time stamped event records in the one or more sets of searchable, time stamped event records includes two or more table portions, and wherein each of the two or more table portions is stored in a distributed manner proximate to a subset of the event records to which it pertains.

16. An apparatus, comprising:

an event record creator, implemented at least partially in hardware, that creates two or more sets of searchable, time stamped event records from raw data stored in at least one data store, wherein each searchable, time stamped event record in the two or more sets of searchable, time stamped event records includes a portion of the raw data and is associated with a time stamp derived from the raw data, wherein the raw data reflects activity in an information technology environment;

a summarization table generator, implemented at least partially in hardware, that generates a summarization table for each set of searchable, time stamped event records in the two or more sets of searchable, time stamped event records that;

identifies one or more field values, wherein a field value comprises a value that appears in an associated field in one or more searchable, time stamped event records in the set of searchable, time stamped event records; and

for each field value, identifies the one or more searchable, time stamped event records in the set of searchable, time stamped event records that contain the field value for the associated field;

a summarization table storage system, implemented at least partially in hardware, that stores the summarization table for each set of searchable, time stamped event records among the two or more sets of time stamped searchable event records;

a summarization table selector, implemented at least partially in hardware, that selects a stored summarization table based on a received query that includes search criteria for evaluating field values for one or more fields;

a subsystem, implemented at least partially in hardware, that uses the search criteria to evaluate field values for one or more fields in the selected summarization table to generate a query result; and

wherein the query result reflects an aspect of activity in the information technology environment.

17. The apparatus of claim 16, further comprising:

a subsystem, implemented at least partially in hardware, that causes display of information based on the query result.

18. The apparatus of claim 16, further comprising:

a subsystem, implemented at least partially in hardware, that causes display of the query result.

19. The apparatus of claim 16, further comprising:

a subsystem, implemented at least partially in hardware, that executes an action based on the query result.

20. The apparatus of claim 16, further comprising:

a subsystem, implemented at least partially in hardware, that based on using the search criteria to evaluate field values for one or more fields in the selected summarization table, retrieves event records identified in the summarization table for further processing to generate the query result.

21. The apparatus of claim 16, wherein the query result is generated from the summarization table with retrieving event records identified in the summarization table.

22. The apparatus of claim 16, further comprising:

a subsystem, implemented at least partially in hardware, that stores the two or more sets of searchable, time stamped event records in an indexed data store.

23. The apparatus of claim 16, further comprising:

a subsystem, implemented at least partially in hardware, that stores the two or more sets of searchable, time stamped event records in an indexed data store;

wherein the indexed data store comprises a distributed indexed data store.

24. The apparatus of claim 16, further comprising:

a subsystem, implemented at least partially in hardware, that stores the two or more sets of searchable, time stamped event records in an indexed data store;

wherein the indexed data store is stored in a distributed manner among two or more servers.

25. The apparatus of claim 16, further comprising:

a subsystem, implemented at least partially in hardware, that receives a command identifying fields to include in the summarization table for each set of searchable, time stamped event records in the two or more sets of searchable, time stamped event records.

26. The apparatus of claim 16, wherein the raw data includes machine data.

27. The apparatus of claim 16, wherein the raw data includes log data.

28. The apparatus of claim 16, wherein the raw data includes unstructured data.

29. The apparatus of claim 16, wherein the summarization table for each set of searchable, time stamped event records in the one or more sets of searchable, time stamped event records includes two or more table portions, and wherein the two or more table portions are stored in a distributed manner.

30. One or more non-transitory computer-readable storage media, storing software instructions, which when executed by one or more processors cause performance of:

creating two or more sets of searchable, time stamped event records from raw data stored in at least one data store, wherein each searchable, time stamped event record in the two or more sets of searchable, time stamped event records includes a portion of the raw data and is associated with a time stamp derived from the raw data, wherein the raw data reflects activity in an information technology environment;

generating a summarization table for each set of searchable, time stamped event records in the two or more sets of searchable, time stamped event records that;

identifies one or more field values, wherein a field value comprises a value that appears in an associated field in one or more searchable, time stamped event records in the set of searchable, time stamped event records; and

for each field value, identifies the one or more searchable, time stamped event records in the set of searchable, time stamped event records that contain the field value for the associated field;

storing the summarization table for each set of searchable, time stamped event records among the two or more sets of time stamped searchable event records;

selecting a stored summarization table based on a received query that includes search criteria for evaluating field values for one or more fields;

using the search criteria to evaluate field values for one or more fields in the selected summarization table to generate a query result; and

wherein the query result reflects an aspect of activity in the information technology environment.