GENERATION OF A SEARCH QUERY TO APPROXIMATE REPLICATION OF A CLUSTER OF EVENTS

• US 20160034525A1
• Filed: 07/31/2014
• Published: 02/04/2016
• Est. Priority Date: 07/31/2014
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Neels, Alice, Zhang, Steve, Robichaud, Marc
• Granted Patent: US 10,296,616 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/242 Query formulation
• CPC Class Codes: G06F 16/285 Clustering or classification

Abstract

A processing device performs a preliminary grouping of data items in a dataset to define one or more clusters and for each cluster, identifies a set of search terms for a search query that would retrieve data items in the cluster upon execution of the search query against the dataset.

Claims

1. A method comprising:

accessing data items in a dataset, each data item containing a portion of raw machine-generated data in textual form generated by a component in an information-technology environment;

applying a clustering algorithm to the data items to group the data items into two or more clusters, the clustering algorithm generating an ordered list of keywords for each data item that is parsed from that data item and grouping data items into a same cluster when their respective ordered lists of keywords meet a similarity threshold; and

for each cluster, identifying a set of one or more search terms providing criteria for a search query that substantially reproduces the cluster upon execution of the search query against the dataset, wherein execution of the search query against the dataset comprises evaluating the search terms against the raw machine-generated data in textual form in the data items;

wherein each of the search terms requires a presence of a particular keyword in the data items, requires an absence of a particular keyword in the data items, or includes a criterion pertaining to a field in the data items;

wherein the method is performed by one or more processing devices.

2. The method of claim 1, wherein the search terms identified for at least one cluster require the presence of the particular keyword in the data items.

3. The method of claim 1, wherein the search terms identified for at least one cluster require the absence of the particular keyword in the data items.

4. The method of claim 1, wherein the search terms identified for at least one cluster require the presence of the particular field-value combination in the data items.

5. The method of claim 1, wherein identifying the set of search terms for the search query for a cluster comprises determining search terms that, when applied to the dataset, produces a set of data items that is close to the data items in the cluster.

6. The method of claim 1, further comprising:

for a particular cluster, testing alternative combinations of search terms to discover one combination that better reproduces the data items in the particular cluster than another combination when applied to the dataset.

7. The method of claim 1, further comprising saving the search query for a particular cluster as an event type.

8. The method of claim 1, further comprising:

saving the search query for a particular cluster as an event type that includes a reference name for the event type;

executing the search query defining the event type; and

tagging data items retrieved by the search query with a tag corresponding to the reference name.

9. The method of claim 1, further comprising:

saving the search query for a particular cluster as an event type that includes a reference name for the event type;

determining that a particular data item that has been displayed to a user satisfies criteria of a search query defining the event type; and

displaying the reference name for the event type in association with information about the particular data item.

10. The method of claim 1, wherein applying the clustering algorithm to the data items includes identifying one or more tokens in each data item, the tokens comprising keywords.

11. The method of claim 1, wherein the clustering algorithm includes

generating a token vector for each of the data items;

and

grouping data items having token vectors within a similarity threshold into a same cluster of the one or more clusters.

12. The method of claim 1, wherein identifying the set of search terms for each cluster comprises identifying one or more tokens included in the data items in the cluster, the tokens comprising keywords.

13. The method of claim 1, wherein identifying the set of search terms for each cluster comprises identifying each of the data items in the dataset that contain a particular token.

14. The method of claim 1, wherein identifying the set of search terms for each cluster comprises determining a percentage of data items that include a given token in each of the one or more clusters.

15. The method of claim 1, wherein identifying the set of search terms for each cluster comprises:

determining a percentage of data items that include a given token in each of the one or more clusters; and

averaging the determined percentages for each of the one or more clusters.

16. The method of claim 1, wherein identifying the set of search terms for each cluster comprises determining a variance, across each of the one or more clusters, in a percentage of data items in the one or more clusters that include a given token.

17. The method of claim 1, wherein identifying the set of search terms for each cluster comprises calculating a relevance score for each token in each of the one or more clusters, wherein the relevance score is based at least in part on a percentage of data items in each of the one or more clusters that include a given token, an average of the percentages for each of the one or more clusters, and a variance in the percentages.

18. The method of claim 1, wherein identifying the set of search terms for a particular cluster comprises identifying a number of tokens having a highest relevance score for the particular cluster.

19. The method of claim 1, further comprising:

causing display of information about at least one of the clusters, a representative data item from the cluster, and a percentage of the dataset included in the cluster.

20. The method of claim 1, further comprising:

causing display of the set of search terms for the search query identified for one of the clusters.

21. The method of claim 1, further comprising:

causing display of a particular search query that includes the search terms identified for one of the clusters.

22. The method of claim 1, further comprising:

causing display of a particular search query or an identifier of the particular search query that includes the search terms identified for one of the clusters;

receiving input indicating that a user wants to execute the particular search; and

based on receiving the input, causing execution of the search query.

23. The method of claim 1, wherein the data items in the dataset are generated by a search language that uses a late binding schema.

24. The method of claim 1, wherein the search query is in a search language that uses a late binding schema.

25. The method of claim 1, wherein the data items in the dataset comprise raw machine data.

26. The method of claim 1, wherein data items in the dataset comprise time-stamped events, the time-stamped events each including at least a portion of raw machine data.

27. A system comprising:

a memory; and

a processing device coupled to the memory, the processing device to;

accessing data items in a dataset, each data item containing a portion of raw machine-generated data in textual form generated by a component in an information-technology environment;

apply a clustering algorithm to the data items to group the data items into two or more clusters, the clustering algorithm generating an ordered list of keywords for each data item that is parsed from that data item and grouping data items into a same cluster when their respective ordered lists of keywords meet a similarity threshold; and

for each cluster, identify a set of one or more search terms providing criteria for a search query that substantially reproduces the cluster upon execution of the search query against the dataset, wherein execution of the search query against the dataset comprises evaluating the search terms against the raw machine-generated data in textual form in the data items;

wherein each of the search terms requires a presence of a particular keyword in the data items, requires an absence of a particular keyword in the data items, or includes a criterion pertaining to a field in the data items.

28. The system of claim 27, wherein the processing device further to:

save the search query for a particular cluster as an event type.

29. A non-transitory computer-readable storage medium storing instructions which, when executed by a processing device, cause the processing device to perform operations comprising:

access data items in a dataset, each data item containing a portion of raw machine-generated data in textual form generated by a component in an information-technology environment;

applying a clustering algorithm to the data items to group the data items into two or more clusters, the clustering algorithm generating an ordered list of keywords for each data item that is parsed from that data item and grouping data items into a same cluster when their respective ordered lists of keywords meet a similarity threshold; and

for each cluster, identifying a set of one or more search terms providing criteria for a search query that substantially reproduces the cluster upon execution of the search query against the dataset, wherein execution of the search query against the dataset comprises evaluating the search terms against the raw machine-generated data in textual form in the data items;

wherein each of the search terms requires a presence of a particular keyword in the data items, requires an absence of a particular keyword in the data items, or includes a criterion pertaining to a field in the data items;

wherein the operations are performed by one or more processing devices.

30. The non-transitory computer-readable storage medium of claim 29, wherein the operations further comprise:

saving the search query for a particular cluster as an event type.