Refining extraction rules based on selected text within events

• US 10,394,946 B2
• Filed: 09/01/2017
• Issued: 08/27/2019
• Est. Priority Date: 09/07/2012
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Miller, Jesse, Delfino, Micah James, Robichaud, Marc, Hanson, Catherine Anne, Carasso, David
• Primary Examiner(s): Phantana-angkool, David
• Application Number:
  US15/694,654
• Publication Number:
  US 20180267947A1
• Time in Patent Office: 725 Days
• Field of Search: 715823
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G06F 40/279 Recognition of textual entities

Abstract

The technology disclosed relates to formulating and refining field extraction rules that are used at query time on raw data with a late-binding schema. The field extraction rules identify portions of the raw data, as well as their data types and hierarchical relationships. These extraction rules are executed against very large data sets not organized into relational structures that have not been processed by standard extraction or transformation methods. By using sample events, a focus on primary and secondary example events help formulate either a single extraction rule spanning multiple data formats, or multiple rules directed to distinct formats. Selection tools mark up the example events to indicate positive examples for the extraction rules, and to identify negative examples to avoid mistaken value selection. The extraction rules can be saved for query-time use, and can be incorporated into a data model for sets and subsets of event data.

Claims

1. A computer-implemented method comprising:

receiving data indicating selection of a first event from among a plurality of events, wherein each event includes a portion of raw data and is associated with a time stamp;

receiving data indicating a selection of one or more portions of text within the first event to be extracted as one or more fields;

automatically determining at least one field extraction rule that extracts one or more values for the one or more fields from the text within the plurality of events when the extraction rule is applied to the plurality of events;

causing display of an annotated version of the plurality of events, wherein the annotated version indicates the portions of the text within the plurality of events extracted by the field extraction rule, the annotated version of the plurality of events including a second event to be used to refine field extraction; and

based on a selection of at least one portion of text within the second event to be extracted, updating the field extraction rule.

2. The method of claim 1, wherein the raw data is from machine data.

3. The method of claim 1, wherein the raw data is from server data.

4. The method of claim 1, further including:

transmitting one or more tools that implement user selection of the one or more portions of text within the first event and naming of the one or more fields.

5. The method of claim 1, further including:

providing tools that implement user selection of a sampling strategy;

receiving a selection of the sampling strategy; and

resampling and updating the events to be displayed.

6. The method of claim 1, further including:

providing tools that implement user selection of a sampling strategy;

receiving a selection of a diverse sampling strategy;

resampling according to the diverse sampling strategy, comprising clustering a set of events into multiple clusters, calculating a size of each cluster, and selecting one or more events from each cluster in a set of larger size clusters; and

updating the events to be displayed.

7. The method of claim 1, further including:

providing tools that implement user selection of a sampling strategy;

receiving a selection of a rare sampling strategy;

resampling according to the rare sampling strategy, comprising clustering a set of events into multiple clusters, calculating a size of each cluster, and selecting one or more events from each cluster in a set of smaller size clusters; and

updating the events to be displayed.

8. The method of claim 1, further including:

providing tools that implement user selection of a sampling strategy;

receiving a selection of a time range sampling strategy;

resampling according to the time range sampling strategy, retrieving at least a sample of events in the selected time range; and

updating the events to be displayed.

9. The method of claim 1, further including:

providing one or more tools to select the at least one portion of text within the second event and to link the selected at least one portion of text to the one or more fields already created.

10. The method of claim 1, further including:

providing one or more tools that implement user selection of events that are non-matches to the field extraction rule;

receiving further data indicating a selection of a match or non-match subset of events;

resampling according to the match or non-match selection; and

updating the events to be displayed.

11. The method of claim 1, further including:

receiving a search specification that identifies events to be selected; and

causing display of a search response interface in which the events are responsive to the search specification, the search response interface further including a user option to initiate formulation of a text extraction rule.

12. The method of claim 1, further including:

automatically determining at least one updated field extraction rule that extracts the one or more values of the one or more fields from both the first event and the second event.

13. The method of claim 1, further including:

automatically determining at least one updated field extraction rule that extracts the one or more values of the one or more fields for both the first event and the second event; and

causing display of a second annotated version of the plurality of events, wherein the second annotated version indicates the portions of text within the plurality of events extracted by the updated field extraction rule.

14. The method of claim 1, further including:

receiving a selection to validate the extraction rule;

causing display of the annotated version of the plurality of events, wherein the annotated version provides one or more user controls that implement user selection of indicated portions of the text as examples of text that should not be extracted;

receiving a selection of one or more examples of text that should not be extracted; and

automatically determining at least one updated field extraction rule that does not extract the text that should not be extracted.

15. The method of claim 1, further including:

providing tools that implement user selection of among the fields;

receiving a selection of a selected field; and

transmitting data for a frequency display of values of the selected field extracted from a sample of the events, wherein the frequency display includes a list of values extracted and for each value in the list a frequency and an active filter control, wherein the active filter control filters events to be displayed based on a selected value.

16. The method of claim 1, further including:

providing tools that implement user selection of a particular field among fields for which extraction rules have been created;

receiving a selection of a selected field; and

transmitting data for a frequency display of values of the selected field extracted from a sample of the events, wherein the frequency display includes a list of values extracted and for each value in the list, frequency information and at least one filter control;

receiving a selection of a selected value from the list of values extracted and activation of the filter control; and

transmitting data for a filtered display of values of the selected field extracted from an event sample filtered by the selected value.

17. The method of claim 1, further including saving the extraction rule in a configuration file for lateruse.

18. The method of claim 1, further including:

receiving a selection to save the extraction rule for later use in processing events; and

incorporating the saved extraction rule in a data model that includes a late binding schema of extraction rules applied at search time.

19. The method of claim 1, further including:

providing one or more tools that implement user entry of a filter value to determine the events in the display;

receiving further data indicating entry of a key-value pair to apply as a filter;

resampling according to the key-value pair; and

updating the events to be displayed.

20. A computer-implemented system comprising:

a processor, memory coupled to the processor, and instructions stored in the memory that implement the actions of;

receiving data indicating selection of a first event from among a plurality of events, wherein each event includes a portion of raw data and is associated with a time stamp;

receiving data indicating a selection of one or more portions of text within the first event to be extracted as one or more fields;

automatically determining at least one field extraction rule that extracts one or more values for the one or more fields from the text within the plurality of events when the extraction rule is applied to the plurality of events;

causing display of an annotated version of the plurality of events, wherein the annotated version indicates the portions of the text within the plurality of events extracted by the field extraction rule, the annotated version of the plurality of events including a second event to be used to refine field extraction; and

based on a selection of at least one portion of text within the second event to be extracted, updating the field extraction rule.

21. The system of claim 20, further including:

providing tools that implement user selection of a sampling strategy to determine the events in the display;

receiving a selection of the sampling strategy; and

resampling and updating the events to be displayed.

22. The system of claim 20, further including:

providing one or more tools to select the at least one portion of text within the second event and to link the selected at least one portion of text to the one or more fields already created.

23. The system of claim 20, further including:

automatically determining at least one updated field extraction rule that extracts as one or more values of the one or more fields from both the first event and the second event.

24. The system of claim 20, further including:

providing tools that implement user selection of among the fields;

receiving a selection of a selected field; and

transmitting data for a frequency display of values of the selected field extracted from a sample of the events, wherein the frequency display includes a list of values extracted and for each value in the list a frequency and an active filter control, wherein the active filter control filters events to be displayed based on a selected value.

25. The system of claim 20, further including:

receiving a selection to save the extraction rule for later use in processing events; and

incorporating the saved extraction rule in a data model that includes a late binding schema of extraction rules applied at search time.

26. A tangible computer-readable memory having instructions stored in the memory that implement the actions including:

receiving data indicating selection of a first event from among a plurality of events, wherein each event includes a portion of raw data and is associated with a time stamp;

receiving data indicating a selection of one or more portions of text within the first event to be extracted as one or more fields;

automatically determining at least one field extraction rule that extracts one or more values for the one or more fields from the text within the plurality of events when the extraction rule is applied to the plurality of events;

causing display of an annotated version of the plurality of events, wherein the annotated version indicates the portions of the text within the plurality of events extracted by the field extraction rule, the annotated version of the plurality of events including a second event to be used to refine field extraction; and

based on a selection of at least one portion of text within the second event to be extracted, updating the field extraction rule.

27. The tangible computer-readable memory of claim 26, further including:

providing tools that implement user selection of a sampling strategy to determine the events in the display;

receiving a selection of the sampling strategy; and

resampling and updating the events to be displayed.

28. The tangible computer-readable memory of claim 26, further including:

providing tools to select the one or more portions of text within the second event and to link the selected portions of text to the one or more fields already created.

29. The tangible computer-readable memory of claim 26, further including:

automatically determining at least one updated field extraction rule that extracts as one or more values of the one or more fields from both the first event and the second event.

30. The tangible computer-readable memory of claim 26, further including:

providing tools that implement user selection among the fields;

receiving a selection of a selected field; and

transmitting data for a frequency display of values of the selected field extracted from a sample of the events, wherein the frequency display includes a list of values extracted and for each value in the list a frequency and an active filter control, wherein the active filter control filters events to be displayed based on a selected value.