Indexing Preview

• US 20140337354A1
• Filed: 07/28/2014
• Published: 11/13/2014
• Est. Priority Date: 08/17/2012
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Blank, Mitchell Neuman Jr., Budchenko, Leonid, Carasso, David, Delfino, Micah James, Hwang, Johnvey, Sorkin, Stephen Phillip, Woo, Eric Timothy
• Granted Patent: US 9,208,206 B2
• Time in Patent Office: Days
• US Class Current: 707/741
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24564 Applying rules; Deductive queries
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application
• CPC Class Codes: G06F 16/316 Indexing structures
• CPC Class Codes: G06F 16/901 Indexing; Data structures therefor; Storage structures for retrieval from the web G06F16/951
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

Embodiments are directed towards previewing results generated from indexing data raw data before the corresponding index data is added to an index store. Raw data may be received from a preview data source. After an initial set of configuration information may be established, the preview data may be submitted to an index processing pipeline. A previewing application may generate preview results based on the preview index data and the configuration information. The preview results may enable previewing how the data is being processed by the indexing application. If the preview results are not acceptable, the configuration information may be modified. The preview application enables modification of the configuration information until the generated preview results may be acceptable. If the configuration information is acceptable, the preview data may be processed and indexed in one or more index stores.

Claims

1. A method, comprising:

selecting a portion of raw data from at least one data store for preview;

parsing the preview portion of raw data into a set of events using one or more parsing rules;

causing display of information from the set of events in a graphical user interface;

in response to user input, processing raw data from the at least one data store using the one or more parsing rules, the raw data including at least some data not in the preview portion of raw data;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, further comprising:

automatically analyzing the preview portion of raw data to select the one or more parsing rules from a plurality of parsing rules.

3. The method of claim 1, further comprising:

automatically analyzing raw data in the at least one data store to find matches of signatures and/or patterns of known data types corresponding to parsing rules in a plurality of parsing rules;

selecting the one or more parsing rules based on any found matches of signatures and/or patterns of the known data types.

4. The method of claim 1, wherein the parsing the preview portion of raw data further comprises:

automatically analyzing the preview portion of raw data to find matches of signatures and/or patterns of known data types corresponding to parsing rules in a plurality of parsing rules;

causing display in a graphical user interface of parsing rules that have been determined to correspond to known data types that match or closely match data in the preview portion of raw data, wherein the determined parsing rules are caused to be displayed in a graphically distinct manner to indicate to a user that the determined parsing rules may be relevant.

5. The method of claim 1, wherein the one or more parsing rules are created by a user.

6. The method of claim 1, wherein the one or more parsing rules are created by a user based on existing parsing rules.

7. The method of claim 1, wherein the display of information from the set of events includes an indication of how at least some of the preview portion of raw data has been parsed.

8. The method of claim 1, further comprising:

prior to parsing the preview portion of raw data into the set of events using the one or more parsing rules, causing display of information from a set of events generated by applying a different set of one or more parsing rules to the preview portion of raw data and then receiving different user input indicating a user preference not to use that different set of one or more parsing rules to process further the raw data from the at least one data store.

9. An apparatus, comprising:

a subsystem, implemented at least partially in hardware, that selects a portion of raw data from at least one data store for preview;

a parsing subsystem, implemented at least partially in hardware, that parses the preview portion of raw data into a set of events using one or more parsing rules;

a subsystem, implemented at least partially in hardware, that causes display of information from the set of events in a graphical user interface;

a subsystem, implemented at least partially in hardware, that, in response to user input, processes raw data from the at least one data store using the one or more parsing rules, the raw data including at least some data not in the preview portion of raw data.

10. The apparatus of claim 9, further comprising:

a subsystem, implemented at least partially in hardware, that automatically analyzes the preview portion of raw data to select the one or more parsing rules from a plurality of parsing rules.

11. The apparatus of claim 9, further comprising:

a subsystem, implemented at least partially in hardware, that automatically analyzes raw data in the at least one data store to find matches of signatures and/or patterns of known data types corresponding to parsing rules in a plurality of parsing rules;

a subsystem, implemented at least partially in hardware, that selects the one or more parsing rules based on any found matches of signatures and/or patterns of the known data types.

12. The apparatus of claim 9, wherein the parsing subsystem further comprises:

a subsystem, implemented at least partially in hardware, that automatically analyzes the preview portion of raw data to find matches of signatures and/or patterns of known data types corresponding to parsing rules in a plurality of parsing rules;

a subsystem, implemented at least partially in hardware, that causes display in a graphical user interface of parsing rules that have been determined to correspond to known data types that match or closely match data in the preview portion of raw data, wherein the determined parsing rules are caused to be displayed in a graphically distinct manner to indicate to a user that the determined parsing rules may be relevant.

13. The apparatus of claim 9, wherein the one or more parsing rules are created by a user.

14. The apparatus of claim 9, wherein the one or more parsing rules are created by a user based on existing parsing rules.

15. The apparatus of claim 9, wherein the display of information from the set of events includes an indication of how at least some of the preview portion of raw data has been parsed.

16. The apparatus of claim 9, further comprising:

a subsystem, implemented at least partially in hardware, that, prior to the parsing subsystem parsing the preview portion of raw data into the set of events using the one or more parsing rules, causes display of information from a set of events generated by applying a different set of one or more parsing rules to the preview portion of raw data and then receiving different user input indicating a user preference not to use that different set of one or more parsing rules to process further the raw data from the at least one data store.

17. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to perform:

selecting a portion of raw data from at least one data store for preview;

parsing the preview portion of raw data into a set of events using one or more parsing rules;

causing display of information from the set of events in a graphical user interface;

in response to user input, processing raw data from the at least one data store using the one or more parsing rules, the raw data including at least some data not in the preview portion of raw data;

wherein the method is performed by one or more computing devices.

18. The non-transitory computer-readable medium of claim 17, further comprising:

automatically analyzing the preview portion of raw data to select the one or more parsing rules from a plurality of parsing rules.

19. The non-transitory computer-readable medium of claim 17, further comprising:

automatically analyzing raw data in the at least one data store to find matches of signatures and/or patterns of known data types corresponding to parsing rules in a plurality of parsing rules;

selecting the one or more parsing rules based on any found matches of signatures and/or patterns of the known data types.

20. The non-transitory computer-readable medium of claim 17, wherein the parsing the preview portion of raw data further comprises:

automatically analyzing the preview portion of raw data to find matches of signatures and/or patterns of known data types corresponding to parsing rules in a plurality of parsing rules;

causing display in a graphical user interface of parsing rules that have been determined to correspond to known data types that match or closely match data in the preview portion of raw data, wherein the determined parsing rules are caused to be displayed in a graphically distinct manner to indicate to a user that the determined parsing rules may be relevant.

21. The non-transitory computer-readable medium of claim 17, wherein the one or more parsing rules are created by a user.

22. The non-transitory computer-readable medium of claim 17, wherein the one or more parsing rules are created by a user based on existing parsing rules.

23. The non-transitory computer-readable medium of claim 17, wherein the display of information from the set of events includes an indication of how at least some of the preview portion of raw data has been parsed.

24. The non-transitory computer-readable medium of claim 17, further comprising:

prior to parsing the preview portion of raw data into the set of events using the one or more parsing rules, causing display of information from a set of events generated by applying a different set of one or more parsing rules to the preview portion of raw data and then receiving different user input indicating a user preference not to use that different set of one or more parsing rules to process further the raw data from the at least one data store.