QUERY HANDLING FOR FIELD SEARCHABLE RAW MACHINE DATA

• US 20170139964A1
• Filed: 01/31/2017
• Published: 05/18/2017
• Est. Priority Date: 05/18/2012
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Sorkin, Stephen Phillip, Blank, JR., Mitchell Neuman, Marquardt, David Ryan
• Granted Patent: US 10,402,384 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/282 Hierarchical databases, e.g. IMS, LDAP data stores or Lotus Notes
• CPC Class Codes: G06F 16/319 Inverted lists
• CPC Class Codes: G06F 16/33 Querying
• CPC Class Codes: G06F 16/338 Presentation of query results

Abstract

Embodiments are directed towards a method for searching data. The method comprises providing an inverted index that comprises at least one record, wherein the at least one record comprises at least one field name and a corresponding at least one field value. The at least one field name and corresponding value are extracted from time-stamped searchable events that are stored in a field searchable datastore and comprise portions of raw data. The at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored. The method further comprises receiving an incoming search query that references a field name and evaluating the incoming search query. Furthermore, responsive to the evaluating, the method comprises determining results for the incoming search query using the field searchable datastore or the inverted index.

Claims

1. A method for searching data, the method comprising:

providing an inverted index that comprises at least one record comprising at least one field name and a corresponding at least one field value extracted from time-stamped searchable events, the time-stamped searchable events comprising portions of raw machine data and stored in a field searchable datastore, wherein the at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored;

receiving an incoming search query that references a field name;

evaluating the incoming search query; and

responsive to the evaluating, determining results for the incoming search query using;

(i) the field searchable datastore;

or (ii) the inverted index.

2. The method of claim 1, further comprising:

indexing the time-stamped searchable events to generate indexed time-stamped searchable events; and

wherein the determining results for the incoming search query using the field searchable datastore comprises searching the indexed time-stamped searchable events to determine the results for the incoming search query.

3. The method of claim 1, wherein the incoming search query comprises keywords and the field name.

4. The method of claim 3, wherein a search for the keywords is serviced using the field searchable data store.

5. The method of claim 1, wherein the incoming search query comprises functions that generate statistics concerning portions of the field searchable datastore.

6. The method of claim 1, wherein the field searchable data store comprises a value for a field referred to by the field name in at least one of the time-stamped searchable events.

7. The method of claim 6, wherein the field searchable data store can be searched using the field name, and further comprising:

responsive to the incoming search query, retrieving at least one of the time-stamped searchable events comprising values associated with the field name.

8. The method of claim 1, wherein the field name in the incoming search query is defined by a regular expression rule, wherein the regular expression rule comprises instructions for parsing a value associated with the field name out of at least one of the time-stamped searchable events.

9. The method of claim 1, wherein the field name in the incoming search query is defined in a configuration file, wherein the configuration file comprises regular expression rules for parsing a value associated with the field name out of the time-stamped searchable events.

10. The method of claim 1, wherein the incoming search query comprises at least one aggregate function, and further comprising:

identifying at least one record in the inverted index corresponding to a field name that is associated with the aggregate function; and

determining a result for the incoming search query using the inverted index, wherein determining the result comprises a calculation using posting values associated with the at least one record.

11. A network device that is operative for searching data, the network device comprising:

a memory that is operative to store at least one instruction; and

a processor device that is operative to execute instructions that enable actions, the actions comprising;

providing an inverted index that comprises at least one record comprising at least one field name and a corresponding at least one field value extracted from time-stamped searchable events, wherein the time-stamped searchable events comprise portions of raw machine data and are stored in a field searchable datastore, wherein said at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored;

receiving an incoming search query that references a field name;

evaluating the incoming search query; and

responsive to the evaluating, determining results for the incoming search query using;

(i) the field searchable datastore;

or (ii) the inverted index.

12. The network device of claim 11, further comprising:

indexing the time-stamped searchable events to generate indexed time-stamped searchable events; and

wherein the determining results for the incoming search query using the field searchable datastore comprises searching the indexed time-stamped searchable events to determine results for the incoming search query.

13. The network device of claim 11, wherein the incoming search query comprises keywords and the field name.

14. The network device of claim 13, wherein a search for the keywords is serviced using the field searchable data store.

15. The network device of claim 11, wherein the incoming search query comprises functions that generate statistics concerning portions of the field searchable datastore.

16. The network device of claim 11, wherein the field searchable data store comprises a value for a field referred to by the field name in at least one of the time-stamped searchable events.

17. The network device of claim 16, wherein the field searchable data store can be searched using the field name, and wherein the actions further comprise:

responsive to the incoming search query, retrieving at least one of the time-stamped searchable events comprising values associated with the field name.

18. The network device of claim 11, wherein the field name in the incoming search query is defined by a regular expression rule, wherein the regular expression rule comprises instructions for parsing a value associated with the field name out of at least one of the time-stamped searchable events.

19. The network device of claim 11, wherein the field name in the incoming search query is defined in a configuration file, wherein the configuration file comprises regular expression rules for parsing a value associated with the field name out of the time-stamped searchable events.

20. The network device of claim 11, wherein the incoming search query comprises at least one aggregate function, and wherein the actions further comprise:

identifying at least one record in the inverted index corresponding to a field name that is associated with the aggregate function; and

determining a result for the incoming search query using the inverted index, wherein determining the result comprises a calculation using posting values associated with the at least one record.

21. A processor readable non-transitive storage media that includes instructions wherein execution of the instructions by a processor device enables actions, wherein the actions comprise:

providing an inverted index that comprises at least one record comprising at least one field name and a corresponding at least one field value extracted from time-stamped searchable events, the time-stamped searchable events comprising portions of raw machine data and stored in a field searchable datastore, wherein the at least one record further comprises a posting value that identifies a location in the field searchable datastore where an event associated with the at least one record is stored;

receiving an incoming search query that references a field name;

evaluating the incoming search query; and

responsive to the evaluating, determining results for the incoming search query using;

(i) the field searchable datastore;

or (ii) the inverted index.

22. The processor of claim 21, further comprising:

indexing the time-stamped searchable events to generate indexed time-stamped searchable events; and

wherein the determining results for the incoming search query using the field searchable datastore comprises searching the indexed time-stamped searchable events to determine results for the incoming search query.

23. The processor of claim 21, wherein the incoming search query comprises keywords and the field name.

24. The processor of claim 23, wherein a search for the keywords is serviced using the field searchable data store.

25. The processor of claim 21, wherein the incoming search query comprises functions that generate statistics concerning portions of the field searchable datastore.

26. The processor of claim 21, wherein the field searchable data store comprises a value for a field referred to by the field name in at least one of the time-stamped searchable events.

27. The processor of claim 26, wherein the field searchable data store can be searched using the field name, and wherein the actions further comprise:

responsive to the incoming search query, retrieving at least one of the time-stamped searchable events comprising values associated with the field name.

28. The processor of claim 21, wherein the field name in the incoming search query is defined by a regular expression rule, wherein the regular expression rule comprises instructions for parsing a value associated with the field name out of at least one of the time-stamped searchable events.

29. The processor of claim 21, wherein the field name in the incoming search query is defined in a configuration file, wherein the configuration file comprises regular expression rules for parsing a value associated with the field name out of the time-stamped searchable events.

30. The processor of claim 21, wherein the incoming search query comprises at least one aggregate function, and wherein the actions further comprise:

identifying at least one record in the inverted index corresponding to a field name that is associated with the aggregate function; and

determining a result for the incoming search query using the inverted index, wherein determining the result comprises a calculation using posting values associated with the at least one record.