ADVANCED FIELD EXTRACTOR WITH MODIFICATION OF AN EXTRACTED FIELD

• US 20170139887A1
• Filed: 01/27/2017
• Published: 05/18/2017
• Est. Priority Date: 09/07/2012
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Miller, Jesse, Carasso, David, Delfino, Micah James, Robichaud, Marc
• Granted Patent: US 10,783,318 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/177 of tables; using ruled lines
• CPC Class Codes: G06F 7/24 Sorting, i.e. extracting data from one or more carriers, rearranging the data in numerical or other ordered sequence, and rerecording the sorted data on the original carrier or on a different carrier or set of carriers sorting methods in generalG06F7/36 takes precedence

Abstract

The technology disclosed relates to formulating and refining field extraction rules that are used at query time on raw data with a late-binding schema. The field extraction rules identify portions of the raw data, as well as their data types and hierarchical relationships. These extraction rules are executed against very large data sets not organized into relational structures that have not been processed by standard extraction or transformation methods. By using sample events, a focus on primary and secondary example events help formulate either a single extraction rule spanning multiple data formats, or multiple rules directed to distinct formats. Selection tools mark up the example events to indicate positive examples for the extraction rules, and to identify negative examples to avoid mistaken value selection. The extraction rules can be saved for query-time use, and can be incorporated into a data model for sets and subsets of event data.

Claims

1. A method, comprising:

receiving a first selection associated with an event of a plurality of events, wherein each event in the plurality of events includes a portion of raw data, and wherein the first selection is of a portion of text within the raw data of the event to be extracted as a value of a field;

automatically determining an extraction rule that extracts the selected portion of text as the value of the field; and

causing display of an interface to allow user modification of a representation of the value.

2. The method of claim 1, wherein the user modification includes a concatenation of the value with additional text within the raw data of the event.

3. The method of claim 1, wherein the user modification includes a concatenation of the value with additional text within the raw data of the event, further comprising:

receiving a second selection of the additional text; and

updating the extraction rule to combine the value with the additional text.

4. The method of claim 1, wherein the user modification includes a second selection of a sub-portion of the portion of text, the sub-portion to be trimmed from the value.

5. The method of claim 1, wherein the user modification includes a second selection of a sub-portion of the portion of text, further comprising updating the extraction rule to trim the sub-portion from the value.

6. The method of claim 1, wherein the user modification includes a second selection of a sub-portion of the portion of text, the method further comprising updating the extraction rule to extract the sub-portion as the value of the field.

7. The method of claim 1, further comprising:

receiving a second selection of a sub-portion of the portion of text;

automatically determining a secondary extraction rule to extract the sub-portion; and

updating the extraction rule to include the secondary extraction rule as the value of the field.

8. The method of claim 1, wherein the modified representation of the value is associated with a second field.

9. The method of claim 1, wherein each of the plurality of events is associated with a time stamp.

10. The method of claim 1, further comprising:

receiving a second selection of a sampling strategy; and

causing display of an annotated version of the plurality of events in response to the second selection.

11. The method of claim 1, further comprising:

receiving a second selection of events that match the extraction rule;

sampling according to the second selection; and

causing display of an annotated version of the events that match based on the second selection.

12. The method of claim 1, further comprising:

receiving a second selection of one or more examples of text that should not be extracted; and

automatically determining an updated extraction rule that does not extract the text that should not be extracted.

13. The method of claim 1, further comprising:

in response to a selection of a selected field, causing display of a frequency table of values of the selected field extracted from a sample of the plurality of events.

14. The method of claim 1, further comprising:

receiving a second selection to save the extraction rule and a field name of the field for later use in processing events; and

incorporating the saved extraction rule and field name in a data model that includes a late-binding schema of extraction rules applied at search time.

15. The method of claim 1, further comprising:

receiving a keyword to apply as a filter;

resampling according to the keyword; and

determining events of the plurality of events to be displayed based on the applied keyword.

16. A computer-implemented system comprising:

receiving a first selection associated with an event of a plurality of events, wherein each event in the plurality of events includes a portion of raw data, and wherein the first selection is of a portion of text within the raw data of the event to be extracted as a value of a field;

automatically determining an extraction rule that extracts the selected portion of text as the value of the field; and

causing display of an interface to allow user modification of a representation of the value.

17. The computer-implemented system of claim 16, wherein the user modification includes a concatenation of the value with additional text within the raw data of the event.

18. The computer-implemented system of claim 16, wherein the user modification includes a second selection of a sub-portion of the portion of text, the sub-portion to be trimmed from the value.

19. The computer-implemented system of claim 16, wherein the user modification includes a second selection of a sub-portion of the portion of text, further comprising updating the extraction rule to extract the sub-portion as the value of the field.

20. The computer-implemented system of claim 16, further comprising:

receiving a second selection of events that match the extraction rule;

sampling according to the second selection; and

causing display of an annotated version of the events that match based on the second selection.

21. The computer-implemented system of claim 16, further comprising:

receiving a second selection of one or more examples of text that should not be extracted; and

automatically determining an updated extraction rule that does not extract the text that should not be extracted.

22. The computer-implemented system of claim 16, further comprising:

receiving a second selection to save the extraction rule and a field name of the field for later use in processing events; and

incorporating the saved extraction rule and field name in a data model that includes a late-binding schema of extraction rules applied at search time.

23. The computer-implemented system of claim 16, further comprising:

receiving a keyword to apply as a filter;

resampling according to the keyword; and

determining events of the plurality of events to be displayed based on the applied keyword.

24. A tangible computer-readable memory having instructions stored in the memory that implement the actions including:

receiving a first selection associated with an event of a plurality of events, wherein each event in the plurality of events includes a portion of raw data, and wherein the first selection is of a portion of text within the raw data of the event to be extracted as a value of a field;

automatically determining an extraction rule that extracts the selected portion of text as the value of the field; and

causing display of an interface to allow user modification of a representation of the value.

25. The tangible computer-readable memory of claim 24, wherein the user modification includes a concatenation of the value with additional text within the raw data of the event.

26. The tangible computer-readable memory of claim 24, wherein the user modification includes a second selection of a sub-portion of the portion of text, the sub-portion to be trimmed from the value.

27. The tangible computer-readable memory of claim 24, wherein the user modification includes a second selection of a sub-portion of the portion of text, the actions further including updating the extraction rule to extract the sub-portion as the value of the field.

28. The tangible computer-readable memory of claim 24, the actions further including:

receiving a second selection of events that match the extraction rule;

sampling according to the second selection; and

causing display of an annotated version of the events that match based on the second selection.

29. The tangible computer-readable memory of claim 24, the actions further including:

receiving a second selection of one or more examples of text that should not be extracted; and

automatically determining an updated extraction rule that does not extract the text that should not be extracted.

30. The tangible computer-readable memory of claim 24, the actions further including:

receiving a second selection to save the extraction rule and a field name of the field for later use in processing events; and

incorporating the saved extraction rule and field name in a data model that includes a late-binding schema of extraction rules applied at search time.