Query handling for field searchable raw machine data and associated inverted indexes

• US 10,423,595 B2
• Filed: 01/31/2017
• Issued: 09/24/2019
• Est. Priority Date: 05/18/2012
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Marquardt, David Ryan, Blank, Jr., Mitchell Neuman, Sorkin, Stephen Phillip
• Primary Examiner(s): Le, Thu Nguyet T
• Application Number:
  US15/421,212
• Publication Number:
  US 20170139965A1
• Time in Patent Office: 966 Days
• Field of Search: 707711, 707741, 707742, 707769
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/282 Hierarchical databases, e.g. IMS, LDAP data stores or Lotus Notes
• CPC Class Codes: G06F 16/319 Inverted lists
• CPC Class Codes: G06F 16/33 Querying
• CPC Class Codes: G06F 16/338 Presentation of query results

Abstract

Embodiments are directed towards a method for searching data. The method comprises providing an inverted index that comprises at least one record, wherein the at least one record comprises at least one field name and a corresponding at least one field value. The at least one field name and corresponding value are extracted from time-stamped searchable events that are stored in a field searchable datastore and comprise portions of raw data. The at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored. The method further comprises receiving an incoming search query that references a field name and evaluating the incoming search query. Furthermore, responsive to the evaluating, the method comprises determining results for the incoming search query using both of the field searchable datastore and the inverted index.

Claims

1. A method for searching data, the method comprising:

providing an inverted index that comprises at least one record comprising at least one field name and a corresponding at least one field value extracted from time-stamped searchable events, the time-stamped searchable events comprising portions of raw machine data and stored in a field searchable datastore, wherein the at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored;

receiving an incoming search query that references a field name, wherein the incoming search query comprises keywords and the field name;

evaluating the incoming search query, wherein the evaluating comprises decomposing the search query to analyze it and determine respective portions of the search query addressable by the field searchable datastore and by the inverted index; and

responsive to the evaluating, determining results for the incoming search query by executing the search query across both the field searchable datastore and the inverted index, wherein the field name in the search query is used to perform a search of the inverted index, wherein a search for the keywords is serviced using the field searchable data store, and values corresponding to the field name are searched for in the inverted index.

2. The method of claim 1, further comprising:

indexing the time-stamped searchable events to generate indexed time-stamped searchable events; and

wherein the determining results for the incoming search query using the field searchable datastore comprises searching the indexed time-stamped searchable events to determine the results for the incoming search query.

3. The method of claim 1, wherein the incoming search query comprises functions that generate statistics concerning portions of the field searchable datastore.

4. The method of claim 1, wherein the incoming search query comprises an aggregate function, wherein a result of the aggregate function is derived from a calculation involving at least one posting value in the inverted index.

5. The method of claim 1, wherein the field searchable data store comprises a value for a field referred to by the field name in at least one of the time-stamped searchable events.

6. The method of claim 5, wherein the field searchable data store can be searched using the field name, and further comprising:

responsive to the incoming search query, retrieving at least one of the time-stamped searchable events comprising values associated with the field name.

7. The method of claim 1, wherein the field name in the incoming search query is defined by a regular expression rule, wherein the regular expression rule comprises instructions for parsing a value associated with the field name out of at least one of the time-stamped searchable events.

8. The method of claim 1, wherein the field name in the incoming search query is defined in a configuration file, wherein the configuration file comprises regular expression rules for parsing a value associated with the field name out of at least one of the time-stamped searchable events.

9. A network device that is operative for searching data, the network device comprising:

a transceiver that is operative to communicate over a network;

a memory that is operative to store at least one instruction; and

a processor device that is operative to execute instructions that enable actions, the actions comprising;

providing an inverted index that comprises at least one record comprising at least one field name and a corresponding at least one field value extracted from time-stamped searchable events, wherein the time-stamped searchable events comprise portions of raw machine data and are stored in a field searchable datastore, wherein said at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored;

receiving an incoming search query that references a field name, wherein the incoming search query comprises keywords and the field name;

evaluating the incoming search query, wherein the evaluating comprises decomposing the search query to analyze it and determine respective portions of the search query addressable by the field searchable datastore and by the inverted index; and

responsive to the evaluating, determining results for the incoming search query by executing the search query across both the field searchable datastore and the inverted index, wherein the field name in the search query is used to perform a search of the inverted index, wherein a search for the keywords is serviced using the field searchable data store, and values corresponding to the field name are searched for in the inverted index.

10. The network device of claim 9, further comprising:

indexing the time-stamped searchable events to generate indexed time-stamped searchable events; and

wherein the determining results for the incoming search query using the field searchable datastore comprises searching the indexed time-stamped searchable events to determine results for the incoming search query.

11. The network device of claim 9, wherein the incoming search query comprises functions that generate statistics concerning portions of the field searchable datastore.

12. The network device of claim 9, wherein the incoming search query comprises an aggregate function, wherein a result of the aggregate function is derived fro a calculation involving at least one posting value in the inverted index.

13. The network device of claim 9, wherein the field searchable data store comprises a value for a field referred to by the field name in at least one of the time-stamped searchable events.

14. The network device of claim 13, wherein the field searchable data store can be searched using the field name, and wherein the actions further comprise:

responsive to the incoming search query, retrieving at least one of the time-stamped searchable events comprising values associated with the field name.

15. The network device of claim 9, wherein the field name in the incoming search query is defined by a regular expression rule, wherein the regular expression rule comprises instructions for parsing a value associated with the field name out of at least one of the time-stamped searchable events.

16. The network device of claim 9, wherein the field name in the incoming search query is defined in a configuration file, wherein the configuration file comprises regular expression rules for parsing a value associated with the field name out of at least one of the time-stamped searchable events.

17. The network device of claim 9, wherein the incoming search query comprises at least one aggregate function, and wherein the actions further comprise:

identifying at least one record in the inverted index corresponding to a field name that is associated with the aggregate function; and

determining a result for the incoming search query using the inverted index, wherein determining the result is determined based on an iteration over field values corresponding to the at least one record.

18. A processor readable non-transitive storage media that includes instructions wherein execution of the instructions by a processor device enables actions, wherein the actions comprise:

providing an inverted index that comprises at least one record comprising at least one field name and a corresponding at least one field value extracted from time-stamped searchable events, the time-stamped searchable events comprising portions of raw machine data and stored in a field searchable datastore, wherein the at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored;

receiving an incoming search query that references a field name, wherein the incoming search query comprises keywords and the field name;

evaluating the incoming search query, wherein the evaluating comprises decomposing the search query to analyze it and determine respective portions of the search query addressable by the field searchable datastore and by the inverted index; and

responsive to the evaluating, determining results for the incoming search query by executing the search query across both the field searchable datastore and the inverted index, wherein the field name in the search query is used to perform a search of the inverted index, wherein a search for the keywords is serviced using the field searchable data store.

19. The processor readable non-transitive storage media of claim 18, wherein the actions further comprise:

indexing the time-stamped searchable events to generate indexed time-stamped searchable events; and

wherein the determining results for the incoming search query using the field searchable datastore comprises searching the indexed time-stamped searchable events to determine results for the incoming search query.

20. The processor readable non-transitive storage media of claim 18, wherein the incoming search query comprises functions that generate statistics concerning portions of the field searchable datastore.

21. The processor readable non-transitive storage media of claim 18, wherein the field searchable data store comprises a value for a field referred to by the field name in at least one of the time-stamped searchable events.

22. The processor readable non-transitive storage media of claim 21, wherein the field searchable data store can be searched using the field name, and wherein the actions further comprise:

responsive to the incoming search query, retrieving at least one of the time-stamped searchable events comprising values associated with the field name.

23. The processor readable non-transitive storage media of claim 18, wherein the field name in the incoming search query is defined by a regular expression rule, wherein the regular expression rule comprises instructions for parsing a value associated with the field name out of at least one of the time-stamped searchable events.

24. The processor readable non-transitive storage media of claim 18, wherein the field name in the incoming search query is defined in a configuration file, wherein the configuration file comprises regular expression rules for parsing a value associated with the field name out of at least one of the time-stamped searchable events.