ADVANCED FIELD EXTRACTOR WITH MODIFICATION OF AN EXTRACTED FIELD

• US 20150154269A1
• Filed: 01/30/2015
• Published: 06/04/2015
• Est. Priority Date: 09/07/2012
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Miller, Jesse, Delfino, Micah James, Robichaud, Marc, Carasso, David
• Granted Patent: US 9,594,814 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/177 of tables; using ruled lines
• CPC Class Codes: G06F 7/24 Sorting, i.e. extracting data from one or more carriers, rearranging the data in numerical or other ordered sequence, and rerecording the sorted data on the original carrier or on a different carrier or set of carriers sorting methods in generalG06F7/36 takes precedence

Abstract

The technology disclosed relates to formulating and refining field extraction rules that are used at query time on raw data with a late-binding schema. The field extraction rules identify portions of the raw data, as well as their data types and hierarchical relationships. These extraction rules are executed against very large data sets not organized into relational structures that have not been processed by standard extraction or transformation methods. By using sample events, a focus on primary and secondary example events help formulate either a single extraction rule spanning multiple data formats, or multiple rules directed to distinct formats. Selection tools mark up the example events to indicate positive examples for the extraction rules, and to identify negative examples to avoid mistaken value selection. The extraction rules can be saved for query-time use, and can be incorporated into a data model for sets and subsets of event data.

Claims

1. A computer-implemented method comprising:

accessing in memory a set of events, each event identified by an associated time stamp;

wherein each event in the set of events includes a portion of raw data;

receiving data indicating selection of a first event from among a first plurality of events and data indicating a selection of one or more portions of text within the raw data of the first event to be extracted as one or more fields;

automatically determining an initial extraction rule that extracts the selected portions of text within the first event;

causing display of a first interface providing tools that implement user modification of the extraction rule, including selecting a field from the one or more fields and;

selecting one or more non-adjoining strings to concatenate with the selected field;

selecting a portion of the selected field to be trimmed from the beginning or end of the selected field;

or

selecting sub-portions of text to extract from within the selected field.

2. The method of claim 1, further including:

receiving further data indicating selection of the one or more non-adjoining strings to concatenate into a concatenated field; and

updating the field extraction rule to combine the non-adjoining strings into the concatenated field.

3. The method of claim 1, further including:

receiving further data indicating one or more trim commands to apply to the selected field; and

updating the field extraction rule to include the trim commands.

4. The method of claim 1, further including:

receiving further data indicating selection of sub-portions of text to extract from within the selected field;

automatically determining a secondary extraction rule to extract the sub-portions of text from within the selected field; and

updating the field extraction rule to include the secondary extraction rule.

5. The method of claim 1, further including:

transmitting for display a second user interface providing tools that implement user selection of a sampling strategy to determine the events in a display;

receiving further data indicating a selection of the sampling strategy;

sampling the events to be displayed; and

transmitting for display a third user interface including an annotated version of the plurality of events, wherein the annotated version indicates the portions of text within the plurality of events extracted by the initial extraction rule.

6. The method of claim 1, further including:

causing display of a second user interface providing tools that implement user selection of only events that match the field extraction rule the field extraction rule;

receiving further data indicating a selection of only the events that match; and

sampling according to the match selection; and

causing display of a third user interface including the plurality of events according to the match selection, wherein the annotated version indicates the portions of text within the plurality of events extracted by the initial extraction rule.

7. The method of claim 1, further including:

receiving further data indicating a selection to validate the extraction rule;

causing display of a second user interface including an annotated version of the plurality of events, wherein the annotated version indicates the portions of text within the plurality of events extracted by the field extraction rule and provides one or more user controls that implement user selection of indicated portions of the text as examples of text that should not be extracted;

receiving further data indicating a selection of one or more examples of text that should not be extracted; and

automatically determining an updated field extraction rule that does not extract the text that should not be extracted.

8. The method of claim 1, further including:

causing display of a second user interface providing tools that implements user selection among the fields;

receiving further data indicating a selection of a selected field; and

causing display of a frequency table of values of the selected field extracted from a sample of the events, wherein the frequency table includes a list of values extracted and for each value in the list a frequency and an active filter control, wherein the active filter control filters events to be displayed based on a selected value.

9. The method of claim 1, further including:

receiving further data indicating a selection to save the extraction rule and field names for later use in processing events; and

incorporating the saved extraction rule and field names in a data model that includes a late binding schema of extraction rules applied at search time.

10. The method of claim 1, further including:

causing display of a second user interface providing one or more tools that implement user entry of a filter value to determine the events in the display;

receiving further data indicating entry of a keyword value to apply as a filter;

resampling according to the keyword value; and

updating the events to be displayed.

11. A computer-implemented system comprising:

a processor, memory coupled to the processor, and instructions stored in the memory that implement the actions of;

accessing in memory a set of events, each event identified by an associated time stamp;

wherein each event in the set of events includes a portion of raw data from machine data;

receiving data indicating selection of a first event from among a first plurality of events and data indicating a selection of one or more portions of text within the raw data of the first event to be extracted as one or more fields;

automatically determining an initial extraction rule that extracts the selected portions of text within the first event;

causing display of a first interface providing tools that implement user modification of the extraction rule, including selecting a field from the one or more fields and;

selecting one or more non-adjoining strings to concatenate with the selected field;

selecting a portion of the selected field to be trimmed from the beginning or end of the selected field;

or

selecting sub-portions of text to extract from within the selected field.

12. The computer-implemented system of claim 11, further including:

receiving further data indicating selection of the one or more non-adjoining strings to concatenate into a concatenated field; and

updating the field extraction rule to combine the non-adjoining strings into the concatenated field.

13. The computer-implemented system of claim 11, further including:

receiving further data indicating one or more trim commands to apply to the selected field; and

updating the field extraction rule to include the trim commands.

14. The computer-implemented system of claim 11, further including:

receiving further data indicating selection of sub-portions of text to extract from within the selected field;

automatically determining a secondary extraction rule to extract the sub-portions of text from within the selected field; and

updating the field extraction rule to include the secondary extraction rule.

15. The computer-implemented system of claim 11, further including:

causing display of a second user interface providing tools that implement user selection of a sampling strategy to determine the events in a display;

receiving further data indicating a selection of the sampling strategy;

sampling the events to be displayed; and

causing display of a third user interface including an annotated version of the plurality of events, wherein the annotated version indicates the portions of text within the plurality of events extracted by the initial extraction rule.

16. The computer-implemented system of claim 11, further including:

causing display of a second user interface providing tools that implement user selection of only events that match the field extraction rule the field extraction rule;

receiving further data indicating a selection of only the events that match; and

sampling according to the match selection; and

causing display of a third user interface including the plurality of events according to the match selection, wherein the annotated version indicates the portions of text within the plurality of events extracted by the initial extraction rule.

17. The computer-implemented system of claim 11, further including:

receiving further data indicating a selection to validate the extraction rule;

causing display of a second user interface including an annotated version of the plurality of events, wherein the annotated version indicates the portions of text within the plurality of events extracted by the field extraction rule and provides one or more user controls that implement user selection of indicated portions of the text as examples of text that should not be extracted;

receiving further data indicating a selection of one or more examples of text that should not be extracted; and

automatically determining an updated field extraction rule that does not extract the text that should not be extracted.

18. The computer-implemented system of claim 11, further including:

causing display of a second user interface providing tools that implement user selection among the fields;

receiving further data indicating a selection of a selected field; and

causing display of a frequency table of values of the selected field extracted from a sample of the events, wherein the frequency table includes a list of values extracted and for each value in the list a frequency and an active filter control, wherein the active filter control filters events to be displayed based on a selected value.

19. The computer-implemented system of claim 11, further including:

receiving further data indicating a selection to save the extraction rule and field names for later use in processing events; and

incorporating the saved extraction rule and field names in a data model that includes a late binding schema of extraction rules applied at search time.

20. The computer-implemented system of claim 11, further including:

causing display of a second user interface providing one or more tools that implement user entry of a filter value to determine the events in the display;

receiving further data indicating entry of a keyword value to apply as a filter; and

resampling according to the keyword value; and

updating the events to be displayed.

21. A tangible computer-readable memory having instructions stored in the memory that implement the actions including:

accessing in memory a set of events, each event identified by an associated time stamp;

wherein each event in the set of events includes a portion of raw data from machine data;

receiving data indicating selection of a first event from among a first plurality of events and data indicating a selection of one or more portions of text within the raw data of the first event to be extracted as one or more fields;

automatically determining an initial extraction rule that extracts the selected portions of text within the first event;

causing display of a first interface providing tools that implement user modification of the extraction rule, including selecting a field from the one or more fields and;

selecting one or more non-adjoining strings to concatenate with the selected field;

selecting a portion of the selected field to be trimmed from the beginning or end of the selected field;

or

selecting sub-portions of text to extract from within the selected field.

22. The tangible computer-readable memory of claim 21, further including:

receiving further data indicating selection of the one or more non-adjoining strings to concatenate into a concatenated field; and

updating the field extraction rule to combine the non-adjoining strings into the concatenated field.

23. The tangible computer-readable memory of claim 21, further including:

receiving further data indicating one or more trim commands to apply to the selected field; and

updating the field extraction rule to include the trim commands.

24. The tangible computer-readable memory of claim 21, further including:

receiving further data indicating selection of sub-portions of text to extract from within the selected field;

automatically determining a secondary extraction rule to extract the sub-portions of text from within the selected field; and

updating the field extraction rule to include the secondary extraction rule.

25. The tangible computer-readable memory of claim 21, further including:

causing display of a second user interface providing tools that implement user selection of a sampling strategy to determine the events in a display;

receiving further data indicating a selection of the sampling strategy;

sampling the events to be displayed; and

causing display of a third user interface including an annotated version of the plurality of events, wherein the annotated version indicates the portions of text within the plurality of events extracted by the initial extraction rule.

26. The tangible computer-readable memory of claim 21, further including:

causing display of a second user interface providing tools that implement user selection of only events that match the field extraction rule the field extraction rule;

receiving further data indicating a selection of only the events that match; and

sampling according to the match selection; and

causing display of a third user interface including the plurality of events according to the match selection, wherein the annotated version indicates the portions of text within the plurality of events extracted by the initial extraction rule.

27. The tangible computer-readable memory of claim 21, further including:

receiving further data indicating a selection to validate the extraction rule;

causing display of a second user interface including an annotated version of the plurality of events, wherein the annotated version indicates the portions of text within the plurality of events extracted by the field extraction rule and provides one or more user controls that implement user selection of indicated portions of the text as examples of text that should not be extracted;

receiving further data indicating a selection of one or more examples of text that should not be extracted; and

automatically determining an updated field extraction rule that does not extract the text that should not be extracted.

28. The tangible computer-readable memory of claim 21, further including:

causing display of a second user interface providing tools that implement user selection among the fields;

receiving further data indicating a selection of a selected field; and

causing display of a frequency table of values of the selected field extracted from a sample of the events, wherein the frequency table includes a list of values extracted and for each value in the list a frequency and an active filter control, wherein the active filter control filters events to be displayed based on a selected value.

29. The tangible computer-readable memory of claim 21, further including:

receiving further data indicating a selection to save the extraction rule and field names for later use in processing events; and

incorporating the saved extraction rule and field names in a data model that includes a late binding schema of extraction rules applied at search time.