Collection query driven generation of inverted index for raw machine data

• US 10,061,807 B2
• Filed: 01/31/2017
• Issued: 08/28/2018
• Est. Priority Date: 05/18/2012
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Marquardt, David Ryan, Blank, Mitchell, Sorkin, Stephen
• Primary Examiner(s): Le, Thu-Nguyet
• Application Number:
  US15/421,236
• Publication Number:
  US 20170139996A1
• Time in Patent Office: 574 Days
• Field of Search: 707711, 707741, 707742
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Embodiments of the present disclosure provide a method for generating an inverted index in accordance with a user generated collection query. The method comprises providing a field searchable data store that comprises a plurality of event records, each event record comprising a time-stamped portion of raw machine data. The method further comprises receiving a collection query that references a field name. Further, responsive to the collection query, an inverted index is generated by: a) determining an extraction rule associated with the field name; b) extracting a field value corresponding to the field name from one or more event records in the field searchable data store using the extraction rule; and c) populating the inverted index responsive to each extracted field value, wherein each entry comprises the field name, the corresponding field value and a reference value that identifies a location in the field searchable data store where an associated event record is stored.

Claims

1. A method for searching data, the method comprising:

providing a field searchable data store comprising a plurality of event records, each event record comprising a time-stamped portion of raw machine data;

receiving a query that comprises a plurality of parts including a collection query, wherein the collection query references a field name, wherein the field name is associated with a location in an event record containing a field value associated with the field name, wherein the collection query is user initiated, and wherein a first part in the plurality of parts is associated with the collection query and executable to generate an inverted index, and wherein one or more additional parts in the plurality of parts are executable for performing additional processing of the data in the inverted index;

responsive to the collection query, generating an inverted index by;

determining an extraction rule associated with the field name;

extracting a field value corresponding to the field name from one or more event records in the field searchable data store using the extraction rule; and

populating the inverted index responsive to each extracted field value, wherein each entry comprises the field name, the corresponding field value and a reference value that identifies a location in the field searchable data store where an associated event record is stored; and

performing the additional processing of the data in the inverted index in accordance with the one or more additional parts in the plurality of parts of the query.

2. The method of claim 1 further comprising:

responsive to an incoming search query that references the field name, retrieving the inverted index and using one or more reference values in the inverted index to identify a set of corresponding event records and retrieving additional information from the set of event records.

3. The method of claim 1, wherein the inverted index is specific to a bucket of data of the field searchable data store, wherein the bucket of data is associated with a specified time frame and comprises a subset of the plurality of event records.

4. The method of claim 1, wherein the field searchable data store is indexed by a plurality of indexers, wherein each indexer indexes a first subset of the plurality of event records, and wherein the inverted index is specific to an indexer.

5. The method of claim 4, wherein each indexer is associated with a plurality of buckets of data, wherein each bucket of data is associated with a specified time frame and comprises a second subset of the plurality of event records, wherein the second subset is a subset of the first subset.

6. The method of claim 1, wherein the extraction rule is retrieved from a configuration file and wherein the extracting a field value comprises executing the extraction rule.

7. The method of claim 6, wherein the extraction rule in the configuration file is associated with metadata specifying a type of event to which the extraction rule applies and a regular expression rule for parsing out a field value associated with the field name.

8. The method of claim 7, wherein the extraction rule is manually entered in the configuration file by a user.

9. The method of claim 7, further comprising:

using a graphical user interface to create the extraction rule in the configuration file, wherein a user highlights portions of a sample event using the graphical user interface to create regular expression rules associated with the highlighted portions.

10. The method of claim 7, wherein the extraction rule is entered in the configuration file using an automatic field discovery process.

11. The method of claim 7, wherein the type of event is a source type.

12. The method of claim 1, further comprising:

updating the inverted index at periodic intervals in accordance with criteria specified in the collection query.

13. The method of claim 1, wherein the one or more additional parts comprises further filtering steps.

14. A method for searching data, the method comprising:

providing a field searchable data store comprising a plurality of event records, each event record comprising a time-stamped portion of raw machine data;

receiving a query that comprises a plurality of parts including a collection query, wherein the collection query references a field name, wherein the field name is associated with a location in an event record containing a field value associated with the field name, wherein the collection query is user initiated, and wherein a first part in the plurality of parts is associated with the collection query and executable to generate an inverted index, and wherein one or more additional parts in the plurality of parts are executable for performing additional processing of the data in the inverted index;

responsive to the collection query, generating an inverted index by;

determining at least one extraction rule, wherein each field name is associated with an extraction rule;

extracting a field value corresponding to each of the at least one field names from one or more event records in the field searchable data store using one of the at least one extraction rules; and

populating the inverted index responsive to each extracted field value, wherein each entry comprises the field name, the corresponding field value and a reference value that identifies a location in the field searchable data store where an associated event record is stored; and

performing the additional processing of the data in the inverted index in accordance with the one or more additional parts in the plurality of parts of the query.

15. The method of claim 14 further comprising:

responsive to an incoming search query that references the field name, retrieving the inverted index and using one or more reference values in the inverted index to identify a set of corresponding event records and retrieving additional information from the set of event records.

16. The method of claim 15, wherein the incoming search query is executed using a pipelined process comprising a first stage in which the inverted index is accessed obtain a set of events.

17. The method of claim 16, wherein the incoming search query further comprises:

a second, subsequent stage executing additional commands for further filtering and processing.

18. The method of claim 17, wherein the incoming search query comprises at least one aggregate function, wherein the at least one aggregate function is in a subsequent stage of the pipelined process, and wherein determining results for the at least one aggregate function comprises:

identifying at least one record in the inverted index corresponding to a field name that is associated with the at least one aggregate function; and

determining a result for the at least one aggregate function using the inverted index, wherein determining the result comprises using a calculation that uses reference values associated with the at least one record.

19. The method of claim 14, wherein each of the at least one extraction rule extraction rules is retrieved from a configuration file and wherein the extracting a field value comprises executing an associated extraction rule.

20. The method of claim 19, wherein each of the at least one extraction rule in the configuration file is associated with metadata specifying a type of event to which the extraction rule applies and a regular expression rule for parsing out a field value associated with the at least one field name.

21. The method of claim 20, wherein the at least one extraction rule is manually entered in the configuration file by a user.

22. A network device that is operative for searching data, the network device comprising:

a transceiver that is operative to communicate over a network;

a memory that is operative to store at least one instruction; and

a processor device that is operative to execute instructions that enable actions, the actions comprising;

providing a field searchable data store comprising a plurality of event records, each event record comprising a time-stamped portion of raw machine data;

receiving a query that comprises a plurality of parts including a collection query, wherein the collection query references a field name, wherein the field name is associated with a location in an event record containing a field value associated with the field name, wherein the collection query is user initiated, and wherein a first part in the plurality of parts is associated with the collection query and executable to generate an inverted index, and wherein one or more additional parts in the plurality of parts are executable for performing additional processing of the data in the inverted index;

responsive to the collection query, generating an inverted index by;

retrieving an extraction rule associated with the field name from a configuration file;

extracting a field value corresponding to the field name from one or more event records in the field searchable data store using the extraction rule; and

populating the inverted index responsive to each extracted field value, wherein each entry comprises the field name, the corresponding field value and a reference value that identifies a location in the field searchable data store where an associated event record is stored; and

performing the additional processing of the data in the inverted index in accordance with the one or more additional parts in the plurality of parts of the query.

23. The network device of claim 22, wherein the actions further comprise:

responsive to an incoming search query that references the field name, retrieving the inverted index and using one or more reference values in the inverted index to identify a set of corresponding event records and retrieving additional information from the set of event records.

24. The network device of claim 22, wherein the inverted index is specific to a bucket of data of the field searchable data store, wherein the bucket of data is associated with a specified time frame and comprises a subset of the plurality of event records.

25. The network device of claim 22, wherein the field searchable data store is indexed by a plurality of indexers, wherein each indexer indexes a first subset of the plurality of event records, and wherein the inverted index is specific to an indexer.

26. The network device of claim 25, wherein each indexer is associated with a plurality of buckets of data, wherein each bucket of data is associated with a specified time frame and comprises a second subset of the plurality of event records, wherein the second subset is a subset of the first subset.

27. The network device of claim 22, wherein the extraction rule in the configuration file is associated with metadata specifying a type of event to which the extraction rule applies and a regular expression rule for parsing out a field value associated with the field name.

28. The network device of claim 27, wherein the extraction rule is manually entered in the configuration file by a user.

29. The network device of claim 27, further comprising:

using a graphical user interface to create the extraction rule in the configuration file, wherein a user highlights portions of a sample event using the graphical user interface to create regular expression rules associated with the highlighted portions.

30. The network device of claim 27, wherein the extraction rule is entered in the configuration file using an automatic field discovery process.

31. The network device of claim 22, wherein the actions further comprise:

updating the inverted index at periodic intervals in accordance with criteria specified in the collection query.