Log data analysis

• US 9,633,106 B1
• Filed: 01/04/2016
• Issued: 04/25/2017
• Est. Priority Date: 06/30/2011
• Status: Active Grant

• Current Assignee: Sumo Logic, Inc.
• Original Assignee: Sumo Logic, Inc.
• Inventors: Saurabh, Kumar, Beedgen, Christian Friedrich, Kurtic, Bruno
• Primary Examiner(s): Wong, Leslie
• Application Number:
  US14/987,064
• Time in Patent Office: 477 Days
• CPC Class Codes: G06F 16/2468 Fuzzy queries
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification

Abstract

Analyzing log data, such as security log data and event data, is disclosed. Log data is obtained. Portions of the log data are clustered into clusters of similar data portions. A signature for each cluster is generated. Comparison of subsequent log data with the signature indicates whether the subsequent log data belongs in the cluster.

Claims

1. A system, comprising:

one or more processors configured to;

obtain log data;

cluster portions of the log data into clusters of similar data portions;

generate a signature for each cluster, wherein the signature comprises a representation of log data in the cluster, wherein comparison of subsequent log data with the signature indicates whether the subsequent log data belongs in the cluster, and wherein the log data in the cluster conforms to the signature; and

cause the generated signature to be presented via one or more interfaces, wherein the one or more interfaces are configured to obtain a user instruction associated with an action to take with respect to the presented signature, wherein the action comprises editing the signature, and wherein the signature is modified in response to the obtained user instruction; and

a memory coupled to the one or more processors and configured to provide the one or more processors with instructions.

2. The system of claim 1 wherein at least a portion of the signature is assigned a label in response to the obtained user instruction.

3. The system of claim 1 wherein the one or more processors are further configured to cause a signature to be hidden from display in response to the obtained user instruction.

4. The system of claim 1 wherein a signature for a given cluster is generated at least in part by determining a print statement used to generate at least some of the log data in the given cluster.

5. The system of claim 1 wherein a signature for a given cluster is generated at least in part by determining static portions of at least some of the log data in the given cluster.

6. The system of claim 5 wherein a signature for a given cluster is generated at least in part by generalizing non-static portions of at least some of the log data in the given cluster.

7. The system of claim 6 wherein the non-static portions are generalized using at least one of tokens and wild cards.

8. The system of claim 1 wherein the one or more or processors are further configured to use a token library to automatically generalize portions of log data in a given cluster.

9. The system of claim 1 wherein the one or more or processors are further configured to cause fields within a given signature that vary to be displayed with wild card placeholders.

10. The system of claim 1 wherein the one or more processors are further configured to replace, for a given signature, tokenized fields with placeholder variables.

11. The system of claim 1 wherein a field of the signature is transformed into a variable.

12. The system of claim 1 wherein the one or more processors are further configured to request confirmation that a signature does not contain confidential information.

13. The system of claim 1 wherein the one or more processors are further configured to request that a user edit a signature to remove confidential information present.

14. The system of claim 1 wherein a field included in the signature has a first level of granularity and wherein the system is configured to, in response to a user request, generate a second signature in which the field has a second level of granularity that is different from the first level of granularity.

15. The system of claim 1 further comprising a storage configured to store the signature in a signature library.

16. The system of claim 15 wherein the storage is configured to store a plurality of signatures in the signature library, wherein the library is accessible by a first and second user, and wherein the first user can use the plurality of signatures in conjunction with data that is not accessible to the second user.

17. The system of claim 15 wherein the signature is stored as a result of actions taken by a first user and wherein the stored signature is modified by a second user.

18. The system of claim 1 wherein the one or more processors are further configured to cause a signature label to be displayed to a user.

19. The system of claim 18 wherein the signature label comprises a representation of the cluster.

20. The system of claim 1 wherein the one or more processors are further configured to cause a set of controls to be displayed to the user.

21. The system of claim 20 wherein at least one control included in the set of controls comprises a control that allows the user to break a cluster into sub-clusters.

22. The system of claim 20 wherein at least one control included in the set of controls comprises a control that allows the user to combine multiple clusters into a single cluster.

23. The system of claim 1 wherein the clustering is performed using nearest neighbor.

24. The system of claim 1 wherein the clustering includes performing a hierarchical clustering using fuzzy matching.

25. The system of claim 24 wherein match scores that exceed a threshold are determined to belong to a same cluster.

26. The system of claim 25 wherein a match score represents a confidence that at least some of the log data was generated using a same print statement.

27. The system of claim 1 wherein the signature is treated as important in response to an obtained user instruction to mark the signature as important.

28. A method, comprising:

obtaining log data;

clustering, using one or more processors, portions of the log data into clusters of similar data portions;

generating a signature for each cluster, wherein the signature comprises a representation of log data in the cluster, wherein comparison of subsequent log data with the signature indicates whether the subsequent log data belongs in the cluster, and wherein the log data in the cluster conforms to the signature; and

causing the generated signature to be presented via one or more interfaces, wherein the one or more interfaces are configured to obtain a user instruction associated with an action to take with respect to the presented signature, wherein the action comprises editing the signature, and wherein the signature is modified in response to the obtained user instruction.

29. The method of claim 28 wherein the signature corresponds to a print statement that was used to generate at least some of the log data.

30. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

obtaining log data;

clustering portions of the log data into clusters of similar data portions;

generating a signature for each cluster, wherein the signature comprises a representation of log data in the cluster, wherein comparison of subsequent log data with the signature indicates whether the subsequent log data belongs in the cluster, and wherein the log data in the cluster conforms to the signature; and

causing the generated signature to be presented via one or more interfaces, wherein the one or more interfaces are configured to obtain a user instruction associated with an action to take with respect to the presented signature, wherein the action comprises editing the signature, and wherein the signature is modified in response to the obtained user instruction.