Supplementing a high performance analytics store with evaluation of individual events to respond to an event query
First Claim
1. A computer implemented method, comprising:
- receiving raw data at a computing device;
parsing the raw data into event records by determining event boundaries in the raw data, wherein each of the event records includes a portion of the raw data and is associated with a time derived from the raw data;
storing the event records in an indexed data store;
generating a summarization table that;
identifies one or more field values, wherein a field value comprises a value that appears in an associated field in one or more of the event records in the indexed data store; and
for each field value, identifies a set of one or more event records in the indexed data store that contain the field value for the associated field;
receiving a query that includes search criteria for evaluating field values for one or more fields;
using the search criteria to evaluate field values for one or more fields in the summarization table to generate a preliminary result set;
determining that the query cannot be answered fully by the summarization table by determining that the indexed data store includes event records that have not been processed for inclusion in the summarization table; and
based on determining that the indexed data store includes event records that have not been processed for inclusion in the summarization table;
using the search criteria to identify supplemental event records in the indexed data store that satisfy the search criteria and that have not been processed for inclusion in the summarization table;
generating a query result using the preliminary result set from the summarization table and the supplemental event records; and
causing display of the query result or transmitting the query result to a second computing device for further processing and output.
1 Assignment
0 Petitions
Accused Products
Abstract
Embodiments are directed are towards the transparent summarization of events. Queries directed towards summarizing and reporting on event records may be received at a search head. Search heads may be associated with one more indexers containing event records. The search head may forward the query to the indexers the can resolve the query for concurrent execution. If a query is a collection query, indexers may generate summarization information based on event records located on the indexers. Event record fields included in the summarization information may be determined based on terms included in the collection query. If a query is a stats query, each indexer may generate a partial result set from previously generated summarization information, returning the partial result sets to the search head. Collection queries may be saved and scheduled to run and periodically update the summarization information.
149 Citations
30 Claims
-
1. A computer implemented method, comprising:
-
receiving raw data at a computing device; parsing the raw data into event records by determining event boundaries in the raw data, wherein each of the event records includes a portion of the raw data and is associated with a time derived from the raw data; storing the event records in an indexed data store; generating a summarization table that; identifies one or more field values, wherein a field value comprises a value that appears in an associated field in one or more of the event records in the indexed data store; and for each field value, identifies a set of one or more event records in the indexed data store that contain the field value for the associated field; receiving a query that includes search criteria for evaluating field values for one or more fields; using the search criteria to evaluate field values for one or more fields in the summarization table to generate a preliminary result set; determining that the query cannot be answered fully by the summarization table by determining that the indexed data store includes event records that have not been processed for inclusion in the summarization table; and based on determining that the indexed data store includes event records that have not been processed for inclusion in the summarization table; using the search criteria to identify supplemental event records in the indexed data store that satisfy the search criteria and that have not been processed for inclusion in the summarization table; generating a query result using the preliminary result set from the summarization table and the supplemental event records; and causing display of the query result or transmitting the query result to a second computing device for further processing and output. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10)
-
-
11. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to perform:
-
receiving raw data at a computing device; parsing the raw data into event records by determining event boundaries in the raw data, wherein each of the event records includes a portion of the raw data and is associated with a time derived from the raw data; storing the event records in an indexed data store; generating a summarization table that; identifies one or more field values, wherein a field value comprises a value that appears in an associated field in one or more of the event records in the indexed data store; and for each field value, identifies a set of one or more event records in the indexed data store that contain the field value for the associated field; receiving a query that includes search criteria for evaluating field values for one or more fields; using the search criteria to evaluate field values for one or more fields in the summarization table to generate a preliminary result set; determining that the query cannot be answered fully by the summarization table by determining that the indexed data store includes event records that have not been processed for inclusion in the summarization table; and based on determining that the indexed data store includes event records that have not been processed for inclusion in the summarization table; using the search criteria to identify supplemental event records in the indexed data store that satisfy the search criteria and that have not been processed for inclusion in the summarization table; generating a query result using the preliminary result set from the summarization table and the supplemental event records; and causing display of the query result or transmitting the query result to a second computing device for further processing and output. - View Dependent Claims (12, 13, 14, 15, 16, 17, 18, 19, 20)
-
-
21. An apparatus, comprising:
-
an indexed data store; a subsystem, implemented at least partially in hardware, that receives raw data at a computing device; a subsystem, implemented at least partially in hardware, that parses the raw data into event records by determining event boundaries in the raw data, wherein each of the event records includes a portion of the raw data and is associated with a time derived from the raw data; a subsystem, implemented at least partially in hardware, that stores the event records in the indexed data store; a subsystem, implemented at least partially in hardware, that generates a summarization table that; identifies one or more field values, wherein a field value comprises a value that appears in an associated field in one or more of the event records in the indexed data store; and for each field value, identifies a set of one or more event records in the indexed data store that contain the field value for the associated field; a subsystem, implemented at least partially in hardware, that receives a query that includes search criteria for evaluating field values for one or more fields; a subsystem, implemented at least partially in hardware that uses the search criteria to evaluate field values for one or more fields in the summarization table to generate a preliminary result set; a query determination subsystem, implemented at least partially in hardware, that determines that the query cannot be answered fully by the summarization table by determining that the indexed data store includes event records that have not been processed for inclusion in the summarization table; and a subsystem, implemented at least partially in hardware that, based on determining that the indexed data store includes event records that have not been processed for inclusion in the summarization table; using the search criteria to identify supplemental event records in the indexed data store that satisfy the search criteria and that have not been processed for inclusion in the summarization table; generating a query result using the preliminary result set from the summarization table and the supplemental event records; and causing display of the query result or transmitting the query result to a second computing device for further processing and output. - View Dependent Claims (22, 23, 24, 25, 26, 27, 28, 29, 30)
-
Specification