Automatic field extraction from filed values

• US 10,726,037 B2
• Filed: 01/30/2015
• Issued: 07/28/2020
• Est. Priority Date: 01/30/2015
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Robichaud, Marc Vincent
• Primary Examiner(s): Bullock, Joshua
• Application Number:
  US14/610,702
• Publication Number:
  US 20160224643A1
• Time in Patent Office: 2,006 Days
• Field of Search: 707602, 707722
• CPC Class Codes: G06F 11/30 Monitoring
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/321 Display for diagnostics, e.g. diagnostic result display, self-test user interface
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing

Abstract

First one or more values are extracted from a plurality of events using a first extraction rule. The extracted first one or more values are assigned to a first field of the plurality of events as a first set of field-data item pairs. Second one or more values are extracted from the plurality of the events using a second extraction rule. The second extraction rule identifies the second one or more values and a field label corresponding to the second one or more values in the extracted first one or more values of the first set of field-data item pairs. The extracted second one or more values are assigned to a second field of the plurality of events as a second set of field-data item pairs. The field label extracted using the second extraction rule or a modified version thereof may be assigned to the second field.

Claims

1. A computer-implemented method, comprising:

accessing a set of events in a data store, each event in the set of events including a portion of raw machine data that reflects activity in an information technology environment and that is produced by a component of that information technology environment;

wherein an extraction rule for extracting a subportion of text from the portion of raw machine data defines a first field for the set of events, the subportion of text containing an extracted value corresponding to the first field;

automatically identifying a second field for the set of events from the first field based on determining that the extracted value for the first field of a first event includes text that corresponds to a field label of the second field and another value associated with the field label of the second field; and

causing display of the field label of the automatically identified second field.

2. The method of claim 1, wherein the determining is based on identifying at least one designated demarcating character separating a first subportion of the text representing the field label of the second field from a second subportion of the text representing the other value associated with the field label of the second field.

3. The method of claim 1, wherein the automatically identifying is part of executing a command that automatically identifies and extracts a plurality of fields for the set of events from the first field, each field being identified by a corresponding field label identified within text of values of the first field.

4. The method of claim 1, wherein the automatically identifying is part of executing a command that automatically identifies the second field, produces values for the second field from values of the first field, and associates the produced values with corresponding events of the set of events.

5. The method of claim 1, wherein the extraction rule comprises instructions to produce the value for the first field when a field label of the first field is identified in the text from the portion of raw machine data in the first event.

6. The method of claim 1, based on the automatically identifying, defining the second field by an additional extraction rule for extracting a subportion of text from the value of the first event to extract the other value for the second field from the first event.

7. The method of claim 1, wherein the second field is defined by an additional extraction rule for extracting a subportion of text from the value of the first field to extract the other value from the second field for the first event based on determining the subportion of text is associated with the field label of the second field in the text of the value of the first field.

8. The method of claim 1, further comprising applying the extraction rule to the set of events to produce values for the first field for the set of events.

9. The method of claim 1, further comprising causing execution of a search query, wherein values of the identified second field are extracted from at least some of the events as part of a command of the search query.

10. The method of claim 1, further comprising:

causing execution of a search query, wherein values of the identified second field are extracted from at least some of the events as part of a command of the search query; and

causing display of at least some of the events in a search interface with at least some of the values of the identified second field.

11. The method of claim 1, wherein the automatically identifying is part of executing a command that causes a new field to be assigned to the set of events for each unique field label of a plurality of unique field labels the command identifies in values of the first field.

12. The method of claim 1, wherein the automatically identifying is part of executing a command that extracts the identified second field from values of the first field, and assigns a field name to the second field that is a modified version of the field label of the second field.

13. The method of claim 1, wherein the automatically identifying is part of executing a command that extracts the identified second field from values of the first field, and assigns a field name to the second field that includes the field label.

14. The method of claim 1, wherein the automatically identifying includes based on identifying an equal sign in the text of the value for the first field, identifying the field label of the second field as a group characters that immediately precedes the equal sign.

15. The method of claim 1, further comprising assigning a modified version of the field label of the second field that was extracted from the first field to the second field as a field name.

16. The method of claim 1, further comprising causing automatic generation of a field name of the second field from the field label of the second field and a field name of the first field.

17. The method of claim 1, comprising applying a late-binding schema to the plurality of events, the late-binding schema extracting values of the first field and the second field using corresponding extraction rules.

18. The method of claim 1, further comprising producing values of the identified second field from at least some of the events in the data store as part of a command of a search query executed by a search system.

19. The method of claim 1, further comprising producing values of the second field from at least some of the events in the data store as part of a first command of a search query executed by a search system, and producing values of the first field from at least some of the events as part of a second command of the search query.

20. The method of claim 1, wherein the extraction rule includes at least one regular expression.

21. A system, comprising:

one or more data processors; and

one or more computer-readable storage media containing instructions which when executed on the one or more data processors, cause the one or more processors to perform operations including;

accessing a set of events in a data store, each event in the set of events including a portion of raw machine data that reflects activity in an information technology environment and that is produced by a component of that information technology environment;

wherein an extraction rule for extracting a subportion of text from the portion of raw machine data defines a first field for the set of events, the subportion of text containing an extracted value corresponding to the first field;

automatically identifying a second field for the set of events from the first field based on determining that the extracted value for the first field includes text that corresponds to a field label of the second field and another value associated with the field label of the second field; and

causing display of the field label of the automatically identified second field.

22. The system of claim 21, wherein the determining is based on identifying at least one designated demarcating character separating a first subportion of the text representing the field label of the second field from a second subportion of the text representing the other value associated with the field label of the second field.

23. The system of claim 21, wherein the automatically identifying is part of executing a command that automatically identifies and extracts a plurality of fields for the set of events from the first field, each field being identified by a corresponding field label identified within text of values of the first field.

24. The system of claim 21, wherein the automatically identifying is part of executing a command that automatically identifies the second field, produces values for the second field from values of the first field, and associates the produced values with corresponding events of the set of events.

25. The system of claim 21, wherein the extraction rule comprises instructions to produce the value for the first field when a field label of the first field is identified in the text from the portion of raw machine data in the first event.

26. One or more non-transitory computer-storage media storing computer-useable instructions that, when executed by a computing device, perform a method, the method comprising:

accessing a set of events in a data store, each event in the set of events including a portion of raw machine data that reflects activity in an information technology environment and that is produced by a component of that information technology environment;

wherein an extraction rule for extracting a subportion of text from the portion of raw machine data defines a first field for the set of events, the subportion of text containing an extracted value corresponding to the first field;

automatically identifying a second field for the set of events from the first field based on determining that the extracted value for the first field of a first event includes text corresponds to a field label of the second field and another value associated with the field label of the second field; and

causing display of the field label of the automatically identified second field.

27. The computer-storage media of claim 26, wherein the determining is based on identifying at least one designated demarcating character separating a first subportion of the text representing the field label of the second field from a second subportion of the text representing the other value associated with the field label of the second field.

28. The computer-storage media of claim 26, wherein the automatically identifying is part of executing a command that automatically identifies and extracts a plurality of fields for the set of events from the first field, each field being identified by a corresponding field label identified within text of values of the first field.

29. The computer-storage media of claim 26, wherein the automatically identifying is part of executing a command that automatically identifies the second field, produces values for the second field from values of the first field, and associates the produced values with corresponding events of the set of events.

30. The computer-storage media of claim 26, wherein the extraction rule comprises instructions to produce the value for the first field when a field label of the first field is identified in the text from the portion of raw machine data in the first event.