Automatic generation of a field-extraction rule based on selections in a sample event

• US 8,909,642 B2
• Filed: 01/23/2013
• Issued: 12/09/2014
• Est. Priority Date: 01/23/2013
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Carasso, R. David, Delfino, Micah James, Hwang, Johnvey
• Primary Examiner(s): Truong, Dennis
• Application Number:
  US13/748,306
• Publication Number:
  US 20140207792A1
• Time in Patent Office: 685 Days
• Field of Search: 707/736, 707/739, 715/808, 715/823, 715/826
• US Class Current: 707/736
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars

Abstract

Embodiments are directed towards automatically generating extraction rules for extracting fields from event records. An extraction rule application receives field data describing the fields to be extracted (including one or more examples) and a collection of event records that may be a representative sample set from a larger set of events records. The extraction rule application generates extraction rules based on the event records and the field data. These extraction rules may be ranked using a determined quality score. Quality scores for extraction rules may be determined based on various metrics related to the operation of the extraction rules and the resultant extracted values. Preferred extraction rules may be determined by ranking the extraction rules based on their quality scores. Also, natural language expressions may be used to create, edit, or modify extraction rules.

Claims

1. A computer-implemented method comprising:

accessing in memory a set of events, each event identified by an associated time stamp;

wherein each event in the set of events includes a portion of raw data from machine data;

transmitting for display a user interface including a first event and a plurality of second events of the set of events;

receiving data indicating a selection of a portion of text within the first event;

automatically determining a field extraction rule that extracts as a value of a field the selection of the portion of text within the first event when the field extraction rule is applied to the first event; and

transmitting for display an updated user interface that includes the second events and that indicates, for each of the second events, a value of the field for each second event that would be extracted by applying the extraction rule to the second event.

5. The method of claim 1, wherein the first event includes machine data.

6. The method of claim 1, wherein the first event includes unstructured data.

7. The method of claim 1, wherein storing the plurality of events includes indexing each event of the plurality of events.

8. The method of claim 1, further comprising:

receiving an indication that a desired value of the field for the second event does not comprise the indicated value of the field for the second event;

modifying the field extraction rule such that the modified field extraction rule would extract the selection of the portion of text when the field extraction rule is applied to the first event and would extract a value of the field for the second event that is different from the indicated value of the field for the second event when the field extraction rule is applied to the second event.

9. The method of claim 1, further comprising:

receiving a selection of a portion of text within the second event;

modifying the field extraction rule such that the modified field extraction rule would extract the selection of the portion of text within the first event when the modified field extraction rule is applied to the first event and would extract the selection of the portion of text within the second event when the modified field extraction rule is applied to the second event.

10. The method of claim 1, wherein the field extraction rule comprises a regular expression.

11. The method of claim 1, further comprising:

displaying natural language representing the field extraction rule;

receiving an edit to the natural language;

determining a modified field extraction rule corresponding to the edited natural language; and

determining values for the field corresponding to the modified field extraction rule.

12. The method of claim 1, further comprising determining a data type of the selection of the portion of text within the first event, and wherein determining the field extraction rule that extracts as the value of the field the selection of the portion of text within the first event comprises determining that the field extraction rule would extract a value comprising the data type as the value of the field for at least one other event.

30. The method of claim 1, further comprising:

receiving an indication that a value is to serve as a counter example for the field; and

modifying the field extraction rule to bias against identifying the counter-example value as a value for the field.

2. A network device that is operative for generating extraction rules, comprising:

including;

a transceiver that is operative to communicate over a network;

a memory that is operative to store at least instructions; and

a processor device that is operative to execute instructions that enable actions, including;

accessing in memory a set of events, each event identified by an associated time stamp;

wherein each event in the set of events includes a portion of raw data from machine data;

transmitting for display a user interface including a first event and a plurality of second events of the set of events;

receiving data indicating a selection of a portion of text within the first event;

automatically determining a field extraction rule that extracts as a value of a field the selection of the portion of text within the first event when the field extraction rule is applied to the first event; and

transmitting for display an updated user interface that includes the second events and that indicates, for each of the second events, a value of the field for each second event that would be extracted by applying the extraction rule to the second event.

13. The network device of claim 2, wherein the first event includes machine data.

14. The network device of claim 2, wherein the first event includes unstructured data.

15. The network device of claim 2, wherein storing the plurality of events includes indexing each event of the plurality of events.

16. The network device of claim 2, wherein the actions further comprise:

receiving an indication that a desired value of the field for the second event does not comprise the indicated value of the field for the second event;

modifying the field extraction rule such that the modified field extraction rule would extract the selection of the portion of text when the modified field extraction rule is applied to the first event and would extract a value of the field for the second event that is different from the indicated value of the field for the second event when the modified field extraction rule is applied to the second event.

17. The network device of claim 2, wherein the actions further comprise:

receiving a selection of a portion of text within the second event;

modifying the field extraction rule such that the modified field extraction rule would extract the selection of the portion of text within the first event when the field extraction rule is applied to the first event and would extract the selection of the portion of text within the second event when the field extraction rule is applied to the second event.

18. The network device of claim 2, wherein the field extraction rule comprises a regular expression.

19. The network device of claim 2, wherein the actions further comprise:

displaying natural language representing the field extraction rule;

receiving an edit to the natural language;

determining a modified field extraction rule corresponding to the edited natural language; and

determining values for the field corresponding to the modified field extraction rule.

20. The network device of claim 2, wherein the actions

further comprise determining a data type of the selection of the portion of text within the first event, and wherein determining the field extraction rule that extracts as the value of the field the selection of the portion of text within the first event comprises determining that the field extraction rule would extract a value comprising the data type as the value of the field for at least one other event.

3. A processor readable non-transitive storage media that includes instructions for generating extraction rules over a network, wherein execution of the instructions by a processor device enables actions, comprising:

accessing in memory a set of events, each event identified by an associated time stamp;

wherein each event in the set of events includes a portion of raw data from machine data;

transmitting for display a user interface including a first event and a plurality of second events of the set of events;

receiving data indicating a selection of a portion of text within the first event;

automatically determining a field extraction rule that extracts as a value of a field the selection of the portion of text within the first event when the field extraction rule is applied to the first event; and

transmitting for display an updated user interface that includes the second events and that indicates, for each of the second events, a value of the field for each second event that would be extracted by applying the extraction rule to the second event.

21. The media of claim 3, wherein the first event includes machine data.

22. The media of claim 3, wherein the first event includes unstructured data.

23. The media of claim 3, wherein storing the plurality of events includes indexing each event of the plurality of events.

24. The media of claim 3, wherein the actions further comprise:

receiving an indication that a desired value of the field for the second event does not comprise the indicated value of the field for the second event;

modifying the field extraction rule such that the modified field extraction rule would extract the selection of the portion of text when the modified field extraction rule is applied to the first event and would extract a value of the field for the second event that is different from the indicated value of the field for the second event when the modified field extraction rule is applied to the second event.

25. The media of claim 3, wherein the actions further comprise:

receiving a selection of a portion of text within the second event;

modifying the field extraction rule such that the modified field extraction rule would extract the selection of the portion of text within the first event when the modified field extraction rule is applied to the first event and would extract the selection of the portion of text within the second event when the modified field extraction rule is applied to the second event.

26. The media of claim 3, wherein the field extraction rule comprises a regular expression.

27. The media of claim 3, wherein the actions further comprise:

displaying natural language representing the field extraction rule;

receiving an edit to the natural language;

determining a modified field extraction rule corresponding to the edited natural language; and

determining values for the field corresponding to the modified field extraction rule.

28. The media of claim 3, wherein the actions further comprise determining a data type of the selection of the portion of text within the first event, and wherein determining the field extraction rule that extracts as the value of the field the selection of the portion of text within the first event comprises determining that the field extraction rule would extract a value comprising the data type as the value of the field for at least one other event.

4. A system that is arranged for generating extraction rules over a network, comprising:

a server device, including;

a transceiver that is operative to communicate over the network;

a memory that is operative to store at least instructions; and

a processor device that is operative to execute instructions that enable actions, including;

accessing in memory a set of events, each event identified by an associated time stamp;

wherein each event in the set of events includes a portion of raw data from machine data;

transmitting for display a user interface including a first event and a plurality of second events of the set of events;

receiving data indicating a selection of a portion of text within the first event;

automatically determining a field extraction rule that extracts as a value of a field the selection of the portion of text within the first event when the field extraction rule is applied to the first event; and

transmitting for display an updated user interface that includes the second events and that indicates, for each of the second events, a value of the field for each second event that would be extracted by applying the extraction rule to the second event.

29. The system of claim 4, wherein the first event includes machine data.