Selecting parsing rules based on data analysis

• US 9,208,206 B2
• Filed: 07/28/2014
• Issued: 12/08/2015
• Est. Priority Date: 08/17/2012
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Blank, Mitchell Neuman Jr., Budchenko, Leonid, Carasso, David, Delfino, Micah James, Hwang, Johnvey, Sorkin, Stephen Phillip, Woo, Eric Timothy
• Primary Examiner(s): Ly, Anh
• Application Number:
  US14/445,001
• Publication Number:
  US 20140337354A1
• Time in Patent Office: 498 Days
• Field of Search: 707/741, 707/716, 707/737, 707/711, 707/723, 707/722, 707/706, 707/756, 707/748, 707/770, 707/769, 707/754, 707/610, 707/602, 707/802, 707/E17.002, 707/E17.014, 707/E17.032, 707/E17.044, 707/E17.108, 709/217, 709/214, 709/244, 709/226, 709/224, 709/218, 709/219, 715/744, 715/727, 715/736, 715/823, 715/826, 715/808, 715/738, 370/399, 370/252, 370/254, 370/255, 370/389, 370/352
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24564 Applying rules; Deductive queries
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application
• CPC Class Codes: G06F 16/316 Indexing structures
• CPC Class Codes: G06F 16/901 Indexing; Data structures therefor; Storage structures for retrieval from the web G06F16/951
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

Embodiments are directed towards previewing results generated from indexing data raw data before the corresponding index data is added to an index store. Raw data may be received from a preview data source. After an initial set of configuration information may be established, the preview data may be submitted to an index processing pipeline. A previewing application may generate preview results based on the preview index data and the configuration information. The preview results may enable previewing how the data is being processed by the indexing application. If the preview results are not acceptable, the configuration information may be modified. The preview application enables modification of the configuration information until the generated preview results may be acceptable. If the configuration information is acceptable, the preview data may be processed and indexed in one or more index stores.

Claims

1. A method, comprising:

selecting a portion of raw data from at least one data source;

analyzing at least the selected portion of raw data to find a match of a signature or pattern of a known data type, the match corresponding to a parsing rule in a plurality of stored parsing rules;

parsing the selected portion of raw data into a set of searchable, time-stamped events using the parsing rule corresponding to the match, each searchable, time-stamped event in the set of searchable, time-stamped events including raw data from the selected portion of raw data;

causing display of a preview of at least a portion of the set of searchable, time-stamped events in a graphical user interface; and

in response to user input received via the graphical user interface, processing raw data from the at least one data source using the parsing rule corresponding to the match, to create searchable, time-stamped events, the processed raw data including at least some data not in the selected portion of raw data;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein the parsing the selected portion of raw data further comprises:

causing display of the parsing rule in the graphical user interface.

3. The method of claim 1, wherein one or more parsing rules in the plurality of stored parsing rules are created by a user.

4. The method of claim 1, wherein one or more parsing rules in the plurality of stored parsing rules are created by a user based on existing parsing rules.

5. The method of claim 1, wherein the display of the preview of the at least a portion of the set of searchable, time-stamped events includes an indication of how at least some of the selected portion of raw data has been parsed.

6. The method of claim 1, further comprising:

prior to parsing the selected portion of raw data into the set of events using the parsing rule, causing display of information from a set of events generated by applying a different parsing rule to the selected portion of raw data and then receiving different user input indicating a user preference not to use that different parsing rule to process further the raw data from the at least one data source.

7. An apparatus, comprising:

a subsystem, implemented at least partially in hardware, that selects a portion of raw data from at least one data source;

a subsystem, implemented at least partially in hardware, that analyzes at least the selected portion of raw data to find a match of a signature or pattern of a known data type, the match corresponding to a parsing rule in a plurality of stored parsing rules;

a parsing subsystem, implemented at least partially in hardware, that parses the selected portion of raw data into a set of searchable, time-stamped events using the parsing rule corresponding to the match, each searchable, time-stamped event in the set of searchable, time-stamped events including raw data from the selected portion of raw data;

a subsystem, implemented at least partially in hardware, that causes display of a preview of at least a portion of the set of searchable, time-stamped events in a graphical user interface; and

a subsystem, implemented at least partially in hardware, that, in response to user input received via the graphical user interface, processes raw data from the at least one data source using the parsing rule corresponding to the match, to create searchable, time-stamped events, the processed raw data including at least some data not in the selected portion of raw data.

8. The apparatus of claim 7, wherein the parsing subsystem further comprises:

a subsystem, implemented at least partially in hardware, that causes display of the parsing rule in the graphical user interface.

9. The apparatus of claim 7, wherein one or more parsing rules in the plurality of stored parsing rules are created by a user.

10. The apparatus of claim 7, wherein one or more parsing rules in the plurality of stored parsing rules are created by a user based on existing parsing rules.

11. The apparatus of claim 7, wherein the display of the preview of the at least a portion of the set of searchable, time-stamped events includes an indication of how at least some of the selected portion of raw data has been parsed.

12. The apparatus of claim 7, further comprising:

a subsystem, implemented at least partially in hardware, that, prior to the parsing subsystem parsing the selected portion of raw data into the set of events using the parsing rules, causes display of information from a set of events generated by applying a different parsing rule to the selected portion of raw data and then receiving different user input indicating a user preference not to use that different parsing rule to process further the raw data from the at least one data source.

13. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to perform:

selecting a portion of raw data from at least one data source;

analyzing at least the selected portion of raw data to find a match of a signature or pattern of a known data type, the match corresponding to a parsing rule in a plurality of stored parsing rules;

parsing the selected portion of raw data into a set of searchable, time-stamped events using the parsing rule corresponding to the match, each searchable, time-stamped event in the set of searchable, time-stamped events including raw data from the selected portion of raw data;

causing display of a preview of at least a portion of the set of searchable, time-stamped events in a graphical user interface; and

in response to user input received via the graphical user interface, processing raw data from the at least one data source using the parsing rule corresponding to the match, to create searchable, time-stamped events, the processed raw data including at least some data not in the selected portion of raw data.

14. The non-transitory computer-readable medium of claim 13, wherein the parsing the preview portion of raw data further comprises:

causing display of the parsing rule in the graphical user interface.

15. The non-transitory computer-readable medium of claim 13, wherein one or more parsing rules in the plurality of stored parsing rules are created by a user.

16. The non-transitory computer-readable medium of claim 13, wherein one or more parsing rules in the plurality of stored parsing rules are created by a user based on existing parsing rules.

17. The non-transitory computer-readable medium of claim 13, wherein the display of the preview of the at least a portion of the set of searchable, time-stamped events includes an indication of how at least some of the selected portion of raw data has been parsed.

18. The non-transitory computer-readable medium of claim 13, wherein the one or more sequences of instructions, when executed by the one or more processors further causes the one or more processors to perform:

prior to parsing the selected portion of raw data into the set of events using the parsing rule, causing display of information from a set of events generated by applying a different parsing rule to the selected portion of raw data and then receiving different user input indicating a user preference not to use that different parsing rule to process further the raw data from the at least one data source.

19. A method, comprising:

selecting a portion of raw data from at least one data source;

parsing the selected portion of raw data into a first set of searchable, time-stamped events using a first parsing rule, each event in the set of searchable events including raw data from the selected portion of raw data;

causing display of a preview of at least a portion of the first set of searchable, time-stamped events in a graphical user interface;

receiving a first user input via the graphical user interface, the first user input indicating a user preference not to use the first parsing rule in indexing raw data associated with the selected portion of raw data;

after receiving the first user input, parsing the selected portion of raw data into a second set of searchable, time-stamped events using a second parsing rule that is different than the first parsing rule, each event in the second set of searchable events including raw data from the selected portion of raw data;

causing display of a preview of at least a portion of the second set of searchable, time-stamped events in the graphical user interface; and

in response to user input received via the graphical user interface, processing raw data from the at least one data source using the second parsing rule to create searchable, time-stamped events, the processed raw data including at least some data not in the selected portion of raw data;

wherein the method is performed by one or more computing devices.

20. The method of claim 19, further comprising:

analyzing the selected portion of raw data to select the first parsing rule from a plurality of parsing rules.

21. The method of claim 19, further comprising:

analyzing raw data from the at least one data source to find matches of signatures or patterns of known data types corresponding to parsing rules in a plurality of parsing rules;

selecting the first parsing rule based on any found matches of signatures and/or patterns of the known data types.

22. The method of claim 19, wherein the parsing the selected portion of raw data further comprises:

analyzing the selected portion of raw data to find matches of signatures or patterns of known data types corresponding to parsing rules in a plurality of parsing rules;

causing display in a graphical user interface of parsing rules that have been determined to correspond to known data types that match or closely match data in the selected portion of raw data, wherein the determined parsing rules are caused to be displayed in a graphically distinct manner to indicate to a user that the determined parsing rules may be relevant.

23. The method of claim 19, wherein the first parsing rule is created by a user.

24. The method of claim 19, wherein the first parsing rule is created by a user based on existing parsing rules.

25. The method of claim 19, wherein the display of the preview of the at least a portion of the set of searchable, time-stamped events includes an indication of how at least some of the selected portion of raw data has been parsed.

26. An apparatus, comprising:

a subsystem, implemented at least partially in hardware, that selects a portion of raw data from at least one data source;

a parsing subsystem, implemented at least partially in hardware, that parses the selected portion of raw data into a first set of searchable, time-stamped events using a first parsing rule, each event in the set of searchable events including raw data from the selected portion of raw data;

a subsystem, implemented at least partially in hardware, that causes display of a preview of at least a portion of the first set of searchable, time-stamped events in a graphical user interface;

a subsystem, implemented at least partially in hardware, that receives a first user input via the graphical user interface, the first user input indicating a user preference not to use the first parsing rule in indexing raw data associated with the selected portion of raw data;

wherein, after receiving the first user input, the parsing subsystem parses the selected portion of raw data into a second set of searchable, time-stamped events using a second parsing rule that is different than the first parsing rule, each event in the second set of searchable events including raw data from the selected portion of raw data;

a subsystem, implemented at least partially in hardware, that causes display of a preview of at least a portion of the second set of searchable, time-stamped events in the graphical user interface; and

a subsystem, implemented at least partially in hardware, that in response to user input received via the graphical user interface, processes raw data from the at least one data source using the second parsing rule to create searchable, time-stamped events, the processed raw data including at least some data not in the selected portion of raw data.

27. The apparatus of claim 26, further comprising:

analyzing the selected portion of raw data to select the first parsing rule from a plurality of parsing rules.

28. The apparatus of claim 26, further comprising:

analyzing raw data from the at least one data source to find matches of signatures or patterns of known data types corresponding to parsing rules in a plurality of parsing rules;

selecting the first parsing rule based on any found matches of signatures and/or patterns of the known data types.

29. The apparatus of claim 26, wherein the parsing subsystem further comprises:

analyzing the selected portion of raw data to find matches of signatures or patterns of known data types corresponding to parsing rules in a plurality of parsing rules;

causing display in a graphical user interface of parsing rules that have been determined to correspond to known data types that match or closely match data in the selected portion of raw data, wherein the determined parsing rules are caused to be displayed in a graphically distinct manner to indicate to a user that the determined parsing rules may be relevant.

30. The apparatus of claim 26, wherein the first parsing rule is created by a user.

31. The apparatus of claim 26, wherein the first parsing rule is created by a user based on existing parsing rules.

32. The apparatus of claim 26, wherein the display of the preview of the at least a portion of the set of searchable, time-stamped events includes an indication of how at least some of the selected portion of raw data has been parsed.

33. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to perform:

selecting a portion of raw data from at least one data source;

parsing the selected portion of raw data into a first set of searchable, time-stamped events using a first parsing rule, each event in the set of searchable events including raw data from the selected portion of raw data;

causing display of a preview of at least a portion of the first set of searchable, time-stamped events in a graphical user interface;

receiving a first user input via the graphical user interface, the first user input indicating a user preference not to use the first parsing rule in indexing raw data associated with the selected portion of raw data;

after receiving the first user input, parsing the selected portion of raw data into a second set of searchable, time-stamped events using a second parsing rule that is different than the first parsing rule, each event in the second set of searchable events including raw data from the selected portion of raw data;

causing display of a preview of at least a portion of the second set of searchable, time-stamped events in the graphical user interface; and

in response to user input received via the graphical user interface, processing raw data from the at least one data source using the second parsing rule to create searchable, time-stamped events, the processed raw data including at least some data not in the selected portion of raw data.

34. The non-transitory computer-readable medium of claim 33, wherein the one or more sequences of instructions, when executed by the one or more processors further causes the one or more processors to perform:

analyzing the selected portion of raw data to select the first parsing rule from a plurality of parsing rules.

35. The non-transitory computer-readable medium of claim 33, wherein the one or more sequences of instructions, when executed by the one or more processors further causes the one or more processors to perform:

analyzing raw data from the at least one data source to find matches of signatures or patterns of known data types corresponding to parsing rules in a plurality of parsing rules;

selecting the first parsing rule based on any found matches of signatures and/or patterns of the known data types.

36. The non-transitory computer-readable medium of claim 33, wherein the parsing the selected portion of raw data further comprises:

analyzing the selected portion of raw data to find matches of signatures or patterns of known data types corresponding to parsing rules in a plurality of parsing rules;

causing display in a graphical user interface of parsing rules that have been determined to correspond to known data types that match or closely match data in the selected portion of raw data, wherein the determined parsing rules are caused to be displayed in a graphically distinct manner to indicate to a user that the determined parsing rules may be relevant.

37. The non-transitory computer-readable medium of claim 33, wherein the first parsing rule is created by a user.

38. The non-transitory computer-readable medium of claim 33, wherein the first parsing rule is created by a user based on existing parsing rules.

39. The non-transitory computer-readable medium of claim 33, wherein the display of the preview of the at least a portion of the set of searchable, time-stamped events includes an indication of how at least some of the selected portion of raw data has been parsed.