COLLECTION QUERY DRIVEN GENERATION OF INVERTED INDEX FOR RAW MACHINE DATA

• US 20170139996A1
• Filed: 01/31/2017
• Published: 05/18/2017
• Est. Priority Date: 05/18/2012
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Sorkin, Stephen, Blank, Mitchell, Marquardt, David Ryan
• Granted Patent: US 10,061,807 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Embodiments of the present disclosure provide a method for generating an inverted index in accordance with a user generated collection query. The method comprises providing a field searchable data store that comprises a plurality of event records, each event record comprising a time-stamped portion of raw machine data. The method further comprises receiving a collection query that references a field name. Further, responsive to the collection query, an inverted index is generated by: a) determining an extraction rule associated with the field name; b) extracting a field value corresponding to the field name from one or more event records in the field searchable data store using the extraction rule; and c) populating the inverted index responsive to each extracted field value, wherein each entry comprises the field name, the corresponding field value and a reference value that identifies a location in the field searchable data store where an associated event record is stored

Claims

1. A method for searching data, the method comprising:

providing a field searchable data store comprising a plurality of event records, each event record comprising a time-stamped portion of raw machine data;

receiving a collection query that references a field name;

responsive to the collection query, generating an inverted index by;

determining an extraction rule associated with the field name;

extracting a field value corresponding to the field name from one or more event records in the field searchable data store using the extraction rule; and

populating the inverted index responsive to each extracted field value, wherein each entry comprises the field name, the corresponding field value and a reference value that identifies a location in the field searchable data store where an associated event record is stored.

2. The method of claim 1 further comprising:

responsive to an incoming search query that references the field name, retrieving the inverted index and using one or more reference values in the inverted index to identify a set of corresponding event records and retrieving additional information from the set of event records.

3. The method of claim 1, wherein the inverted index is specific to a bucket of data of the field searchable data store, wherein the bucket of data is associated with a specified time frame and comprises a subset of the plurality of event records.

4. The method of claim 1, wherein the field searchable data store is indexed by a plurality of indexers, wherein each indexer indexes a first subset of the plurality of event records, and wherein the inverted index is specific to an indexer.

5. The method of claim 4, wherein each indexer is associated with a plurality of buckets of data, wherein each bucket of data is associated with a specified time frame and comprises a second subset of the plurality of event records, wherein the second subset is a subset of the first subset.

6. The method of claim 1, wherein the extraction rule is retrieved from a configuration file and wherein the extracting a field value comprises executing the extraction rule.

7. The method of claim 6, wherein the extraction rule in the configuration file is associated with metadata specifying a type of event to which the extraction rule applies and a regular expression rule for parsing out a field value associated with the field name.

8. The method of claim 7, wherein the extraction rule is manually entered in the configuration file by a user.

9. The method of claim 7, further comprising:

using a graphical user interface to create the extraction rule in the configuration file, wherein a user highlights portions of a sample event using the graphical user interface to create regular expression rules associated with the highlighted portions.

10. The method of claim 7, wherein the extraction rule is entered in the configuration file using an automatic field discovery process.

11. The method of claim 7, wherein the type of event is a source type.

12. The method of claim 1, further comprising:

updating the inverted index at periodic intervals in accordance with criteria specified in the collection query.

30. The method of claim 1, further comprising:

updating the inverted index at periodic intervals in accordance with criteria specified in the collection query.

13. A method for searching data, the method comprising:

providing a field searchable data store comprising a plurality of event records, each event record comprising a time-stamped portion of raw machine data;

receiving a collection query that references at least one field name;

responsive to the collection query, generating an inverted index by;

determining at least one extraction rule, wherein each field name is associated with an extraction rule;

extracting a field value corresponding to each of the at least one field names from one or more event records in the field searchable data store using one of the at least one extraction rules; and

populating the inverted index responsive to each extracted field value, wherein each entry comprises the field name, the corresponding field value and a reference value that identifies a location in the field searchable data store where an associated event record is stored.

14. The method of claim 13 further comprising:

responsive to an incoming search query that references the field name, retrieving the inverted index and using one or more reference values in the inverted index to identify a set of corresponding event records and retrieving additional information from the set of event records.

15. The method of claim 14, wherein the incoming search query is executed using a pipelined process comprising a first stage in which the inverted index is accessed obtain a set of events.

16. The method of claim 15, wherein the incoming search query further comprises:

a second, subsequent stage executing additional commands for further filtering and processing.

17. The method of claim 16, wherein the incoming search query comprises at least one aggregate function, wherein the at least one aggregate function is in a subsequent stage of the pipelined process, and wherein determining results for the at least one aggregate function comprises:

identifying at least one record in the inverted index corresponding to a field name that is associated with the at least one aggregate function; and

determining a result for the at least one aggregate function using the inverted index, wherein determining the result comprises using a calculation that uses reference values associated with the at least one record.

18. The method of claim 13, wherein each of the at least one extraction rule extraction rules is retrieved from a configuration file and wherein the extracting a field value comprises executing an associated extraction rule.

19. The method of claim 18, wherein each of the at least one extraction rule in the configuration file is associated with metadata specifying a type of event to which the extraction rule applies and a regular expression rule for parsing out a field value associated with the at least one field name.

20. The method of claim 19, wherein the at least one extraction rule is manually entered in the configuration file by a user.

21. A network device that is operative for searching data, the network device comprising:

a transceiver that is operative to communicate over a network;

a memory that is operative to store at least one instruction; and

a processor device that is operative to execute instructions that enable actions, the actions comprising;

providing a field searchable data store comprising a plurality of event records, each event record comprising a time-stamped portion of raw machine data;

receiving a collection query that references a field name;

responsive to the collection query, generating an inverted index by;

retrieving an extraction rule associated with the field name from a configuration file;

extracting a field value corresponding to the field name from one or more event records in the field searchable data store using the extraction rule; and

populating the inverted index responsive to each extracted field value, wherein each entry comprises the field name, the corresponding field value and a reference value that identifies a location in the field searchable data store where an associated event record is stored.

22. The network device of claim 21, wherein the actions further comprise:

responsive to an incoming search query that references the field name, retrieving the inverted index and using one or more reference values in the inverted index to identify a set of corresponding event records and retrieving additional information from the set of event records.

23. The network device of claim 21, wherein the inverted index is specific to a bucket of data of the field searchable data store, wherein the bucket of data is associated with a specified time frame and comprises a subset of the plurality of event records.

24. The network device of claim 21, wherein the field searchable data store is indexed by a plurality of indexers, wherein each indexer indexes a first subset of the plurality of event records, and wherein the inverted index is specific to an indexer.

25. The network device of claim 24, wherein each indexer is associated with a plurality of buckets of data, wherein each bucket of data is associated with a specified time frame and comprises a second subset of the plurality of event records, wherein the second subset is a subset of the first subset.

26. The network device of claim 21, wherein the extraction rule in the configuration file is associated with metadata specifying a type of event to which the extraction rule applies and a regular expression rule for parsing out a field value associated with the field name.

27. The network device of claim 26, wherein the extraction rule is manually entered in the configuration file by a user.

28. The network device of claim 26, further comprising:

using a graphical user interface to create the extraction rule in the configuration file, wherein a user highlights portions of a sample event using the graphical user interface to create regular expression rules associated with the highlighted portions.

29. The network device of claim 26, wherein the extraction rule is entered in the configuration file using an automatic field discovery process.