USING AN INVERTED INDEX IN A PIPELINED SEARCH QUERY TO DETERMINE A SET OF EVENT DATA THAT IS FURTHER LIMITED BY FILTERING AND/OR PROCESSING OF SUBSEQUENT QUERY PIPESTAGES

• US 20180218037A1
• Filed: 01/31/2017
• Published: 08/02/2018
• Est. Priority Date: 01/31/2017
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Marquardt, David Ryan, Sabhanatarajan, Karthikeyan, Zhang, Steve Yu
• Granted Patent: US 10,474,674 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24537 of operators
• CPC Class Codes: G06F 16/2477 Temporal data queries

Abstract

Embodiments of the present disclosure provide techniques for using an inverted index in a pipelined search query. A field searchable data store is provided that comprises a plurality of event records, each event record comprising a time-stamped portion of raw machine data. Responsive to the reciept of an incoming search query, the search engine accesses an inverted index, wherein each entry in the inverted index comprises at least one field name, a corresponding at least one field value and a reference value associated with each field name and value pair that identifies a location in the data store where an associated event record is stored. Once the inverted index is accessed, it can be used to filter out a subset of the plurality of event records, wherein the subset comprises one or more event records with corresponding reference values in the inverted index.

Claims

1. A method for searching data, the method comprising:

providing a field searchable data store comprising a plurality of event records, each event record comprising a time-stamped portion of raw machine data;

receiving an incoming query;

generating results for the incoming query using a pipelined process, the pipelined process comprising;

responsive to the incoming query accessing an inverted index, wherein each entry in the inverted index comprises at least one field name, a corresponding at least one field value and a reference value associated with each field name and field value pair that identifies a location in the field searchable data store where an associated event record is stored; and

in a first pipe-stage of the pipelined process, using the inverted index to filter out a first subset of the plurality of event records, wherein the subset comprises one or more event records with corresponding reference values in the inverted index.

2. The method of claim 1, wherein the accessing is in response to an explicit user-specified command in the incoming query.

3. The method of claim 1, wherein the accessing is in response to an automatic determination that using the inverted index would accelerate the incoming query.

4. The method of claim 1, wherein the incoming query is created using a pipelined search language, and further wherein the generating results further comprises:

in a subsequent pipe-stage of the pipelined process, extracting additional information from the first subset of the plurality of event records filtered out by the inverted index in response to additional commands in the subsequent pipe-stage.

5. The method of claim 4, wherein the additional commands in the subsequent pipe-stage are directed to conducting further filtering and processing of the additional information extracted from the first subset of the plurality of event records.

6. The method of claim 1, wherein the inverted index is specific to a bucket of data, wherein the bucket of data is associated with a specified time frame.

7. The method of claim 1, wherein the inverted index is specific to an indexer directed to executing the incoming query on a second subset of the plurality of event records.

8. The method of claim 7, wherein the inverted index is configured to update at periodic intervals in accordance with criteria specified in the incoming query.

9. The method of claim 8, wherein responsive to the incoming query executing before the inverted index updates, the indexer is configured to supplement the search results to the incoming query by extracting data from the second subset of the plurality of event records.

10. A method for searching data, the method comprising:

providing a field searchable data store comprising a plurality of event records, each event record comprising a time-stamped portion of raw machine data;

receiving a collection query that references a field name;

responsive to the collection query, generating an inverted index by;

determining an extraction rule associated with the field name;

extracting a field value corresponding to the field name from one or more event records in the field searchable data store using the extraction rule; and

generating an entry in the inverted index for each of the one or more event records, wherein each entry comprises the field name, the corresponding field value and and a reference value that identifies a location in the field searchable data store where an associated event record is stored; and

responsive to an incoming search query, retrieving the inverted index and using one or more reference values in the inverted index to retrieve additional information from the one or more event records.

11. The method of claim 10, wherein the inverted index is specific to a bucket of data, wherein the bucket of data is associated with a specified time frame.

12. The method of claim 10, wherein the extraction rule is retrieved from a configuration file.

13. The method of claim 10, further comprising:

filtering the additional information retrieved from the one or more event records in accordance with criteria specified in the additional commands.

15. The method of claim 10, wherein the incoming search query is developed using a pipelined process, wherein an output of an inverted index retrieval step can be pipelined to additional commands for further filtering and processing.

16. The method of claim 15, wherein the inverted index is retrieved in a first stage of the pipelined process.

17. The method of claim 16, further comprising:

processing the additional information retrieved from the one or more event records in accordance with criteria specified in the additional commands in a subsequent stage of the pipelined process..

18. The method of claim 17, wherein the processing is performed in response to an aggregate function.

19. The method of claim 10, wherein the inverted index is configured to update at periodic intervals in accordance with criteria specified in the incoming query.

14. The method of claim 14, wherein the filtering is performed in accordance with a conditional command.

20. A system comprising:

a field searchable data store comprising a plurality of event records, each event record comprising a time-stamped portion of raw machine data that is field searchable;

a processing device coupled with the field searchable data store, the processing device being configured to;

receive an incoming query;

responsive to the incoming query, accessing an inverted index, wherein each entry in the inverted index comprises at least one field name, a corresponding at least one field value and a reference value associated with each field name and field value pair that identifies a location in the field searchable data store where an associated event record is stored;

using a pipelined process to generate results of the incoming query, the pipelined process comprising;

in a first pipe stage of the pipeline process, using the inverted index to filter out a first subset of the plurality of event records, wherein the subset comprises one or more event records with corresponding reference values in the inverted index satisfying a portion of the incoming query; and

in a subsequent pipe stage of the pipeline process, performing a field search through raw machine data associated with the first subset of the plurality of event records to obtain field results wherein the field search is performed based on field information specified in the incoming query.

21. The system of claim 20, wherein the incoming query comprises an explicit user-specified command directing access to the inverted index.

22. The system of claim 20, wherein the accessing of the inverted index is performed in response to an automatic determination that using the inverted index accelerates the incoming query.

23. The method of claim 20, wherein the inverted index is specific to a bucket of data, wherein the bucket of data is associated with a specified time frame.

24. The method of claim 20, wherein the field search through the raw machine data is conducted in accordance with an extraction rule specified in a configuration file.

25. The method of claim 20, wherein the inverted index is configured to update at periodic intervals in accordance with criteria specified in the incoming query.

26. A system comprising:

a field searchable data store comprising a plurality of event records, each event record comprising a time-stamped portion of raw machine data that is field searchable;

a processing device coupled with the field searchable data store, the processing device being configured to;

receive an incoming query;

responsive to the incoming query, accessing an inverted index, wherein each entry in the inverted index comprises at least one field name, a corresponding at least one field value and a reference value associated with each field name and field value pair that identifies a location in the field searchable data store where an associated event record is stored;

using a pipelined process to generate results of the incoming query, the pipelined process comprising;

in a first pipe stage of the pipeline process, using the inverted index to filter out a first subset of the plurality of event records, wherein the subset comprises one or more event records with corresponding reference values in the inverted index satisfying a portion of the incoming query; and

in a subsequent pipe stage of the pipeline process, performing a keyword search through raw machine data associated with the first subset of the plurality of event records to obtain keyword search results wherein the keyword search is performed based on information specified in the incoming query.

27. The system of claim 26, wherein the incoming query comprises an explicit user-specified command directing access to the inverted index.

28. The system of claim 26, wherein the accessing of the inverted index is performed in response to an automatic determination that using the inverted index accelerates the incoming query.

29. The method of claim 26, wherein the inverted index is specific to a bucket of data, wherein the bucket of data is associated with a specified time frame.

30. The method of claim 26, wherein the inverted index is configured to update at periodic intervals in accordance with criteria specified in the incoming query.