Generation of a search query to approximate replication of a cluster of events

• US 10,296,616 B2
• Filed: 07/31/2014
• Issued: 05/21/2019
• Est. Priority Date: 07/31/2014
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Neels, Alice, Zhang, Steve, Robichaud, Marc
• Primary Examiner(s): Peach, Polina G
• Application Number:
  US14/449,051
• Publication Number:
  US 20160034525A1
• Time in Patent Office: 1,755 Days
• Field of Search: None
• CPC Class Codes: G06F 16/242 Query formulation
• CPC Class Codes: G06F 16/285 Clustering or classification

Abstract

A processing device performs a preliminary grouping of data items in a dataset to define one or more clusters and for each cluster, identifies a set of search terms for a search query that would retrieve data items in the cluster upon execution of the search query against the dataset.

Claims

1. A method for improving time-based searching of data, the method comprising:

(a) receiving a first user input that specifies a first search query;

(b) in response to the first user input that specifies the first search query,

(b)(1) accessing events in a field-searchable data store, each event being associated with a time stamp and containing a portion of raw machine-generated data indicative of performance or operation of a component in an information-technology environment, wherein the raw machine-generated data contained in each event includes a plurality of strings associated with a corresponding time stamp, and wherein each string includes text, numbers, or a combination of text and numbers;

(b)(2) applying a clustering algorithm to the accessed events to form at least one cluster of events, the clustering algorithm generating, for each event, an ordered list of keywords contained in the event, and grouping events into a same cluster when their respective ordered lists of keywords meet a similarity threshold, and wherein an ordering of the keywords in the ordered list of keywords for any particular event is based on positions of the keywords within the particular event;

(b)(3) for each cluster of the at least one cluster of events, after formation of the cluster, defining, from contents of the cluster, a set of one or more search terms as criteria for a corresponding search query that will retrieve at least one of the events of the cluster when executed against the accessed events, and storing the search terms in memory, wherein identifying the set of one or more search terms for a cluster of events is not part of forming a cluster of events, execution of the corresponding search query against the accessed events including evaluation of the search terms against the raw machine-generated data in the accessed events; and

(b)(4) causing a display of a result of the first search query, the displayed result including an identification of each cluster of the at least one cluster of events;

(c) receiving a second user input that selects a first cluster of the at least one cluster of events, the second user input being responsive to display of the identification of each cluster of the at least one cluster of events; and

(d) in response to the second user input that selects the first cluster,

(d)(1) executing a second search query against the field-searchable data store to retrieve stored events that satisfy a criterion for similarity to the first cluster, the second search query including search terms previously associated with the first cluster as part of said defining, wherein executing the second search query includes retrieving from the memory the search terms previously associated with the first cluster; and

(d)(2) causing display, to the user, of a result of the second search query, including causing display of an event that satisfies the second search query;

wherein the method is performed by one or more processing devices.

2. The method of claim 1, wherein the search terms identified for at least one cluster require the presence of a particular keyword in the events.

3. The method of claim 1, wherein the search terms identified for at least one cluster require the absence of a particular keyword in the events.

4. The method of claim 1, wherein the search terms identified for at least one cluster require the presence of a particular field-value combination in the events.

5. The method of claim 1, wherein identifying the set of search terms for the search query for a cluster comprises determining search terms that, when applied to the field-searchable data store, produce a set of events that includes each of the events in the cluster.

6. The method of claim 1, further comprising:

for a particular cluster, testing alternative combinations of search terms to discover one combination that better reproduces the events in the particular cluster than another combination when applied to the field-searchable data store.

7. The method of claim 1, further comprising saving the search query for a particular cluster as an event type.

8. The method of claim 1, further comprising:

saving the search query for a particular cluster as an event type that includes a reference name for the event type;

executing the search query defining the event type; and

tagging events retrieved by the search query with a tag corresponding to the reference name.

9. The method of claim 1, further comprising:

saving the search query for a particular cluster as an event type that includes a reference name for the event type;

determining that a particular event that has been displayed to a user satisfies criteria of a search query defining the event type; and

displaying the reference name for the event type in association with information about the particular event.

10. The method of claim 1, wherein applying the clustering algorithm to the events includes identifying one or more tokens in each event, the tokens comprising keywords.

11. The method of claim 1, wherein the clustering algorithm includes

generating a token vector for each of the events;

and

grouping events having token vectors within a similarity threshold into a same cluster of the one or more clusters.

12. The method of claim 1, wherein identifying the set of search terms for each cluster comprises identifying one or more tokens included in the events in the cluster, the tokens comprising keywords.

13. The method of claim 1, wherein identifying the set of search terms for each cluster comprises identifying each of the events that contain a particular token.

14. The method of claim 1, wherein identifying the set of search terms for each cluster comprises determining a percentage of events that include a given token in each of the one or more clusters.

15. The method of claim 1, wherein identifying the set of search terms for each cluster comprises:

determining a percentage of events that include a given token in each of the one or more clusters; and

averaging the determined percentages for each of the one or more clusters.

16. The method of claim 1, wherein identifying the set of search terms for each cluster comprises determining a variance, across each of the one or more clusters, in a percentage of events in the one or more clusters that include a given token.

17. The method of claim 1, wherein identifying the set of search terms for each cluster comprises calculating a relevance score for each token in each of the one or more clusters, wherein the relevance score is based at least in part on a percentage of events in each of the one or more clusters that include a given token, an average of the percentages for each of the one or more clusters, and a variance in the percentages.

18. The method of claim 1, wherein identifying the set of search terms for a particular cluster comprises identifying a number of tokens having a highest relevance score for the particular cluster.

19. The method of claim 1, further comprising:

causing display of information about at least one of the clusters, a representative event from the cluster, and a percentage of a dataset included in the cluster.

20. The method of claim 1, further comprising:

causing display of the set of search terms for the search query identified for one of the clusters.

21. The method of claim 1, further comprising:

causing display of a particular search query that includes the search terms identified for one of the clusters.

22. The method of claim 1, further comprising:

causing display of a particular search query or an identifier of the particular search query that includes the search terms identified for one of the clusters;

receiving input indicating that a user wants to execute the particular search; and

based on receiving the input, causing execution of the search query.

23. The method of claim 1, wherein the events are accessed using a search language that uses a late binding schema.

24. The method of claim 1, wherein the search query is in a search language that uses a late binding schema.

25. The method of claim 1, wherein the events comprise time-stamped events.

30. The method of claim 1, wherein identifying a set of one or more search terms is performed without user input after the clustering algorithm groups the events into the same cluster.

31. The method of claim 1, wherein each of the search terms requires at least one of:

a presence of a particular keyword in the events, an absence of a particular keyword in the events, or meeting a criterion pertaining to a field in the events.

26. A system for improving time-based searching of data, the system comprising:

a memory; and

a processing device coupled to the memory, the processing device configured to;

(a) receive a first user input that specifies a first search query;

(b) in response to the first user input that specifies the first search query,

(b)(1) access events in a field-searchable data store, each event being associated with a time stamp and containing a portion of raw machine-generated data indicative of performance or operation of a component in an information-technology environment, wherein the raw machine-generated data contained in each event includes a plurality of strings associated with a corresponding time stamp, and wherein each string includes text, numbers, or a combination of text and numbers;

(b)(2) apply a clustering algorithm to the accessed events to form at least one cluster of events, the clustering algorithm generating, for each event, an ordered list of keywords contained in the event, and grouping events into a same cluster when their respective ordered lists of keywords meet a similarity threshold, and wherein an ordering of the keywords in the ordered list of keywords for any particular event is based on positions of the keywords within the particular event;

(b)(3) for each cluster of the at least one cluster of events, after formation of the cluster, define, from contents of the cluster, a set of one or more search terms as criteria for a corresponding search query that will retrieve at least one of the events of the cluster when executed against the accessed events, and store the search terms in a data store, wherein identifying the set of one or more search terms is not part of forming a cluster of events, execution of the corresponding search query against the accessed events including evaluation of the search terms against the raw machine-generated data in the accessed events; and

(b)(4) cause a display of a result of the first search query, the displayed result including an identification of each cluster of the at least one cluster of events;

(c) receive a second user input that selects a first cluster of the at least one cluster of events, the second user input being responsive to display of the identification of each cluster of the at least one cluster of events; and

(d) in response to the second user input that selects the first cluster,

(d)(1) execute a second search query against the field-searchable data store to retrieve stored events that satisfy a criterion for similarity to the first cluster, the second search query including search terms previously associated with the first cluster as part of said defining, wherein execution of the second search query includes retrieving from the data store the search terms previously associated with the first cluster; and

(d)(2) cause display, to the user, of a result of the second search query, including causing display of an event that satisfies the second search query.

27. The system of claim 26, wherein the processing device further to:

save the search query for a particular cluster as an event type.

32. The system of claim 26, wherein each of the search terms requires at least one of:

a presence of a particular keyword in the events, requires an absence of a particular keyword in the events, or meeting a criterion pertaining to a field in the events.

33. The non-transitory computer-readable storage medium of claim 27, wherein each of the search terms requires at least one of:

a presence of a particular keyword in the events, requires an absence of a particular keyword in the events, or meeting a criterion pertaining to a field in the events.

28. A non-transitory computer-readable storage medium storing instructions execution of which by a processing device causes the processing device to perform operations for improving time-based searching of data, the operations comprising:

(a) receiving a first user input that specifies a first search query;

(b) in response to the first user input that specifies the first search query,

(b)(1) accessing events in a field-searchable data store, each event being associated with a time stamp and containing a portion of raw machine-generated data indicative of performance or operation of a component in an information-technology environment, wherein the raw machine-generated data contained in each event includes a plurality of strings associated with a corresponding time stamp, and wherein each string includes text, numbers, or a combination of text and numbers;

(b)(2) applying a clustering algorithm to the accessed events to form at least one cluster of events, the clustering algorithm generating, for each event, an ordered list of keywords contained in the event, and grouping events into a same cluster when their respective ordered lists of keywords meet a similarity threshold, and wherein an ordering of the keywords in the ordered list of keywords for any particular event is based on positions of the keywords within the particular event;

(b)(3) for each cluster of the at least one cluster of events, after formation of the cluster, defining, from contents of the cluster, a set of one or more search terms as criteria for a corresponding search query that will retrieve at least one of the events of the cluster when executed against the accessed events, and storing the search terms in memory, wherein said identifying the set of one or more search terms is not part of forming a cluster of events, execution of the corresponding search query against the accessed events including evaluation of the search terms against the raw machine-generated data in the events; and

(b)(4) causing a display of a result of the first search query, the displayed result including an identification of each cluster of the at least one cluster of events;

(c) receiving a second user input that selects a first cluster of the at least one cluster of events, the second user input being responsive to display of the identification of each cluster of the at least one cluster of events; and

(d) in response to the second user input that selects the first cluster,

(d)(1) executing a second search query against the field-searchable data store to retrieve stored events that satisfy a criterion for similarity to the first cluster, the second search query including search terms previously associated with the first cluster as part of said defining, wherein executing the second search query includes retrieving from the memory the search terms previously associated with the first cluster; and

(d)(2) causing display, to the user, of a result of the second search query, including causing display of an event that satisfies the second search query;

wherein the operations are performed by one or more processing devices.

29. The non-transitory computer-readable storage medium of claim 28, wherein the operations further comprise:

saving the search query for a particular cluster as an event type.

34. A computer-implemented method for improving time-based searching of data, the method comprising:

(a) accessing events in a field-searchable data store, each event being timestamped and containing a portion of raw machine-generated data indicative of performance or operation of a component in an information-technology environment, wherein the raw machine-generated data contained in each event includes a plurality of strings associated with a corresponding time stamp, and wherein each string includes text, numbers, or a combination of text and numbers;

(b) applying a clustering algorithm to the accessed events to form at least one cluster of events;

(c) for each cluster of the at least one cluster of events,

(c)(1) define a set of one or more search terms for a search query designed to retrieve one or more events in the cluster, and

(c)(2) saving in memory the search query as an event type that includes a reference name, the event type being indicative of a subset of said events;

(d) causing a display, to a user, of the reference name of a particular event type corresponding to a particular cluster;

(e) receiving a user input that selects the particular event type corresponding to the particular cluster, the user input being responsive to the display of the reference name; and

(f) in response to the user input that selects the particular event type corresponding to the particular cluster,

(f)(1) executing a particular search query from the memory to retrieve stored events that satisfy a criterion for similarity to the particular cluster, the particular search query including search terms previously associated with the particular cluster, and

(f)(2) causing display, to the user, of a result of the particular search query, including causing display of an event that satisfies the particular search query.