Directly field searchable and indirectly searchable by inverted indexes raw machine datastore

• US 10,409,794 B2
• Filed: 01/31/2017
• Issued: 09/10/2019
• Est. Priority Date: 05/18/2012
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Marquardt, David Ryan, Blank, Jr., Mitchell Neuman, Sorkin, Stephen Phillip
• Primary Examiner(s): Le, Thu Nguyet T
• Application Number:
  US15/421,297
• Publication Number:
  US 20170140013A1
• Time in Patent Office: 952 Days
• Field of Search: 707711, 707741, 707742, 707769
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/282 Hierarchical databases, e.g. IMS, LDAP data stores or Lotus Notes
• CPC Class Codes: G06F 16/319 Inverted lists
• CPC Class Codes: G06F 16/33 Querying
• CPC Class Codes: G06F 16/338 Presentation of query results

Abstract

Embodiments are directed towards a method for searching data. The method comprises generating an inverted index that comprises at least one record, wherein the at least one record comprises at least one field name and a corresponding at least one field value. The at least one field name and corresponding value are extracted from time-stamped searchable events that are stored in a field searchable datastore and comprise portions of raw data. The at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored. The method further comprises receiving an incoming search query that references a field name. Furthermore, the method comprises generating results to the incoming search query based on the field searchable datastore, wherein the field searchable datastore is directly searchable by the field name.

Claims

1. A method for searching data, the method comprising:

generating an inverted index that comprises at least one record comprising at least one field name and a corresponding at least one field value extracted from time-stamped searchable events comprising portions of raw machine data and stored in a field searchable datastore, wherein each of the at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored;

receiving an incoming search query that references a field name from the at least one field name, wherein the incoming search query comprises keywords and the field name;

evaluating the incoming search query, wherein the evaluating comprises decomposing the search query and determine respective portions of the search query directly addressable by the field searchable datastore and portions for which the field searchable datastore is searchable by entries corresponding to the field name in the inverted index; and

generating results for the incoming search query based on the field searchable datastore, wherein the field searchable datastore is directly searchable by the field name or searchable by entries corresponding to the field name in the inverted index, wherein posting values corresponding to the entries in the inverted index are used to access the field searchable datastore, and further wherein a search for the keywords is directly addressed using the field searchable data store.

2. The method of claim 1, wherein the generating results for the incoming search query based on the field searchable datastore comprises:

indexing the time-stamped searchable events to generate indexed time-stamped searchable events; and

searching the indexed time-stamped searchable events to determine results for the incoming search query.

3. The method of claim 1, wherein the field searchable data store comprises a value for a field referred to by the field name in at least one of the time-stamped searchable events.

4. The method of claim 3, wherein searching the field searchable datastore further comprises:

responsive to the incoming search query, retrieving at least one of the time-stamped searchable events comprising values associated with the field name from the field searchable data store.

5. The method of claim 1, wherein the field name in the incoming search query is identified by using a regular expression rule, wherein the regular expression rule comprises instructions for parsing a value associated with the field name out of the time-stamped searchable events.

6. The method of claim 1, wherein the field name in the incoming search query is defined in a configuration file, wherein the configuration file comprises regular expression rules for parsing a value associated with the field name out of the time-stamped searchable events.

7. The method of claim 1, wherein the incoming search query comprises at least one aggregate function, and wherein the generating results further comprises:

identifying at least one record in the inverted index corresponding to a field name that is associated with the aggregate function; and

determining a result for the incoming search query using the inverted index, wherein determining the result comprises using a calculation that uses posting values associated with the at least one record.

8. A network device that is operative for searching data, the device comprising:

a transceiver that is operative to communicate over a network;

a memory that is operative to store at least one instruction; and

a processor device that is operative to execute instructions that enable actions, the actions comprising;

generating an inverted index that comprises at least one record comprising

at least one field name and a corresponding at least one field value extracted from time-stamped searchable events comprising portions of raw machine data and stored in a field searchable datastore, wherein each of the at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored;

receiving an incoming search query that references a field name from the at least one field name, wherein the incoming search query comprises keywords and the field name;

evaluating the incoming search query, wherein the evaluating comprises decomposing the search query and determine respective portions of the search query directly addressable by the field searchable datastore and portions for which the field searchable datastore is searchable by entries corresponding to the field name in the inverted index; and

generating results for the incoming search query based on the field searchable datastore, wherein the field searchable datastore is directly searchable by the field name or searchable by entries corresponding to the field name in the inverted index, wherein posting values corresponding to the entries in the inverted index are used to access the field searchable datastore, and further wherein a search for the keywords is directly addressed using the field searchable data store.

9. The network device of claim 8, wherein the generating results for the incoming search query using the field searchable datastore comprises:

indexing the time-stamped searchable events to generate indexed time-stamped searchable events; and

searching the indexed time-stamped searchable events to determine results for the incoming search query.

10. The network device of claim 8, wherein the field searchable data store comprises a value for a field referred to by the field name in at least one of the time-stamped searchable events.

11. The network device of claim 10, wherein the actions further comprise:

responsive to the incoming search query, retrieving at least one of the time-stamped searchable events comprising values associated with the field name from the field searchable data store.

12. The network device of claim 8, wherein the field name in the incoming search query is identified by using a regular expression rule, wherein the regular expression rule comprises instructions for parsing a value associated with the field name out of the time-stamped searchable events.

13. The network device of claim 8, wherein the field name in the incoming search query is defined in a configuration file, wherein configuration file comprises regular expression rules for parsing a value associated with the field name out of the time-stamped searchable events.

14. The network device of claim 8, wherein the incoming search query comprises at least one aggregate function, and wherein the generating results further comprises:

identifying at least one record in the inverted index corresponding to a field name that is associated with the aggregate function; and

determining a result for the incoming search query using the inverted index, wherein determining the result comprises using a calculation that uses posting values associated with the at least one record.

15. A processor readable non-transitive storage media that includes instructions wherein execution of the instructions by a processor device enables actions, wherein the actions comprise:

generating an inverted index that comprises at least one record comprising at least one field name and a corresponding at least one field value extracted from time-stamped searchable events comprising portions of raw machine data and stored in a field searchable datastore, wherein each of the at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored;

receiving an incoming search query that references a field name from the at least one field name, wherein the incoming search query comprises keywords and the field name;

evaluating the incoming search query, wherein the evaluating comprises decomposing the search query and determine respective portions of the search query directly addressable by the field searchable datastore and portions for which the field searchable datastore is searchable by entries corresponding to the field name in the inverted index; and

generating results for the incoming search query based on the field searchable datastore, wherein the field searchable datastore is directly searchable by the field name or searchable by entries corresponding to the field name in the inverted index, wherein posting values corresponding to the entries in the inverted index are used to access the field searchable datastore, and further wherein a search for the keywords is directly addressed using the field searchable data store.

16. The processor readable non-transitive storage media of claim 15, wherein the generating results for the incoming search query using the field searchable datastore comprises:

indexing the time-stamped searchable events to generate indexed time-stamped searchable events; and

searching the indexed time-stamped searchable events to determine results for the incoming search query.

17. The processor readable non-transitive storage media of claim 15, wherein the field searchable data store comprises a value for a field referred to by the field name in at least one of the time-stamped searchable events.

18. The processor readable non-transitive storage media of claim 17, wherein the actions further comprise:

responsive to the incoming search query, retrieving at least one of the time-stamped searchable events comprising values associated with the field name from the field searchable data store.

19. The processor readable non-transitive storage media of claim 15, wherein the field name in the incoming search query is identified by using a regular expression rule, wherein the regular expression rule comprises instructions for parsing a value associated with the field name out of the time-stamped searchable events.

20. The processor readable non-transitive storage media of claim 15, wherein the field name in the incoming search query is defined in a configuration file, wherein configuration file comprises regular expression rules for parsing a value associated with the field name out of the time-stamped searchable events.

21. The processor readable non-transitive storage media of claim 15, wherein the incoming search query comprises at least one aggregate function, and wherein the generating results further comprises:

identifying at least one record in the inverted index corresponding to a field name that is associated with the aggregate function; and

determining a result for the incoming search query using the inverted index, wherein determining the result comprises using a calculation that uses posting values associated with the at least one record.