SAMPLING OF EVENTS TO USE FOR DEVELOPING A FIELD-EXTRACTION RULE FOR A FIELD TO USE IN EVENT SEARCHING

• US 20140207784A1
• Filed: 01/30/2014
• Published: 07/24/2014
• Est. Priority Date: 01/22/2013
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: CARASSO, R. David, Delfino, Micah James
• Granted Patent: US 9,031,955 B2
• Time in Patent Office: Days
• US Class Current: 707/739
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/287 Visualization; Browsing
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 7/24 Sorting, i.e. extracting data from one or more carriers, rearranging the data in numerical or other ordered sequence, and rerecording the sorted data on the original carrier or on a different carrier or set of carriers sorting methods in generalG06F7/36 takes precedence

Abstract

Embodiments are directed towards generating a representative sampling as a subset from a larger dataset that includes unstructured data. A graphical user interface enables a user to provide various data selection parameters, including specifying a data source and one or more subset types desired, including one or more of latest records, earliest records, diverse records, outlier records, and/or random records. Diverse and/or outlier subset types may be obtained by generating clusters from an initial selection of records obtained from the larger dataset. An iteration analysis is performed to determine whether a sufficient number of clusters and/or cluster types have been generated that exceed at least one threshold and when not exceeded, additional clustering is performed on additional records. From the resultant clusters, and/or other subtype results, a subset of records is obtained as the representative sampling subset.

Claims

1-30. -30. (canceled)

31. A computer-implemented method, comprising:

receiving machine data at a computing device;

generating a plurality of events, wherein each event in the plurality of events includes a portion of the machine data;

associating a time with each event in the plurality of events, the time for each event extracted from the machine data included in that event;

storing the plurality of events in a data store such that they are searchable at least by their associated times;

selecting a set of events from the plurality of events using a procedure to identify diverse events, a procedure to identify outlier events, a procedure to identify events associated with relatively early times compared to other events in the plurality of events, a procedure to identify events with relatively late times compared to other events in the plurality of events, or a procedure randomly to identify events in the plurality of events;

displaying one or more events in the set of events in a graphical user interface that enables development of a field-extraction rule that specifies how to extract, from the machine data included in each of the one or more events, a value for a field that is defined for each of the one or more events, wherein each of the one or more events is searchable using the field.

32. The method of claim 31, wherein selecting the set of events from the plurality of events includes using a procedure to identify diverse events.

33. The method of claim 31, wherein selecting the set of events from the plurality of events includes using a procedure to identify outlier events.

34. The method of claim 31, wherein selecting the set of events from the plurality of events includes using a procedure to identify events associated with relatively early times compared to other events in the plurality of events.

35. The method of claim 31, wherein selecting the set of events from the plurality of events includes using a procedure to identify events associated with relatively late times compared to other events in the plurality of events.

36. The method of claim 31, wherein selecting the set of events from the plurality of events includes using a procedure randomly to identify events in the plurality of events.

37. The method of claim 31, wherein selecting a set of events from the plurality of events includes using a combination of two or more procedures selected from a procedure to identify diverse events, a procedure to identify outlier events, a procedure to identify events associated with relatively early times compared to other events in the plurality of events, a procedure to identify events with relatively late times compared to other events in the plurality of events, and a procedure randomly to identify events in the plurality of events.

38. The method of claim 31, wherein selecting the set of events from the plurality of events includes using a procedure to identify diverse events, and wherein the procedure to identify diverse events includes:

performing a clustering algorithm on a group of events from the plurality of events to form a plurality of clusters, the clustering algorithm placing two events into a same cluster based on similarities in the machine data included in each of the two events; and

selecting representative events from two or more most populous clusters in the plurality of clusters.

39. The method of claim 31, wherein selecting the set of events from the plurality of events includes using a procedure to identify outlier events, and wherein the procedure to identify outlier events includes:

performing a clustering algorithm on a group of events from the plurality of events to form a plurality of clusters, the clustering algorithm placing two events into a same cluster based on similarities in the machine data included in each of the two events; and

selecting representative events from two or more least populous clusters in the plurality of clusters.

40. The method of claim 31, wherein selecting a set of events from the plurality of events includes using a procedure to identify diverse events or using a procedure to identify outlier events, and wherein the procedure includes:

clustering a group of events in the plurality of events to form a plurality of clusters;

determining that a number of clusters in the plurality of clusters is not big enough; and

clustering a larger group of events in the plurality of events than the group of events.

41. The method of claim 31, wherein selecting a set of events from the plurality of events includes using a procedure to identify diverse events or using a procedure to identify outlier events, and wherein the procedure includes:

clustering a group of events in the plurality of events to form a plurality of clusters;

determining that a number of events in one of the clusters in the plurality of clusters is not big enough; and

clustering a larger group of events in the plurality of events than the group of events.

42. A non-transitory, computer-readable medium having computer-executable instructions for performing the method of claim 31.

43. A computer system with one or more processors adapted to perform the method of claim 31.