Log data analysis

• US 9,262,519 B1
• Filed: 06/05/2012
• Issued: 02/16/2016
• Est. Priority Date: 06/30/2011
• Status: Active Grant

• Current Assignee: Sumo Logic, Sumo Logic, Inc.
• Original Assignee: Sumo Logic, Inc.
• Inventors: Saurabh, Kumar, Beedgen, Christian Friedrich, Kurtic, Bruno
• Primary Examiner(s): Wong, Leslie
• Application Number:
  US13/489,267
• Time in Patent Office: 1,351 Days
• Field of Search: 345/520, 345/522
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2468 Fuzzy queries
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification

Abstract

Analyzing log data, such as security log data and event data, is disclosed. Log data is received. Portions of the log data are clustered into clusters of similar data portions. A signature for each cluster is generated. Comparison of subsequent log data with the signature indicates whether the subsequent log data belongs in the cluster.

Claims

1. A system, comprising:

a first interface configured to receive log data;

a set of one or more processors;

a memory coupled with the set of one or more processors, wherein the memory is configured to provide the set of one or more processors with instructions which when executed cause the set of one or more processors to;

analyze the received log data at least in part by;

clustering portions of the log data into clusters of similar data portions; and

subsequent to the clustering, dynamically generating a signature for each cluster, wherein comparison of subsequent log data with the signature indicates whether the subsequent log data belongs in the cluster; and

cause information associated with the analysis to be displayed to a user;

a storage configured to store a plurality of signatures including the dynamically generated signature in a signature library, wherein the library is accessible by a first and second user, and wherein the first user can use the plurality of signatures in conjunction with data that is not accessible to the second user; and

a second interface configured to receive user input associated with taking action with respect to the displayed information.

2. The system of claim 1 wherein the information caused to be displayed to the user includes the signature.

3. The system of claim 1 wherein the user input received via the second interface includes an instruction from the user to modify the generated signature.

4. The system of claim 1 wherein the user input received via the second interface includes an instruction from a user to label at least a portion of the signature.

5. The system of claim 1 wherein the user input received via the second interface includes an instruction from the user to treat the signature as important.

6. The system of claim 1 wherein the user input received via the second interface includes an instruction from the user to hide the signature.

7. The system of claim 1 wherein the signature corresponds to a print statement that was used to generate at least some of the log data.

8. The system of claim 1 wherein a field included in the signature has a first level of granularity and wherein the set of one or more processors is further configured to, in response to a user request, generate a second signature in which the field has a second level of granularity that is different from the first level of granularity.

9. The system of claim 1 wherein the information caused to be displayed to the user includes a representation of at least a portion of the log data.

10. The system of claim 9 wherein the representation comprises a representation of the cluster.

11. The system of claim 1 wherein the set of one or more processors is further configured to cause a set of controls to be displayed to the user.

12. The system of claim 11 wherein at least one control included in the set of controls comprises a control that allows the user to break a cluster into sub-clusters.

13. The system of claim 11 wherein at least one control included in the set of controls comprises a control that allows the user to combine multiple clusters into a single cluster.

14. A method, comprising:

receiving, via a first interface, log data;

analyzing, using a set of one or more processors, the received log data at least in part by;

clustering portions of the log data into clusters of similar data portions; and

subsequent to the clustering, dynamically generating a signature for each cluster, wherein comparison of subsequent log data with the signature indicates whether the subsequent log data belongs in the cluster;

causing, using the set of one or more processors, information associated with the analysis to be displayed to a user;

storing a plurality of signatures including the dynamically generated signature in a signature library, wherein the library is accessible by a first and second user, and wherein the first user can use the plurality of signatures in conjunction with data that is not accessible to the second user; and

receiving, via a second interface, user input associated with taking action with respect to the displayed information.

15. The method of claim 14, wherein the user input received via the second interface includes an instruction from the user to modify the generated signature.

16. The method of claim 14 wherein the signature corresponds to a print statement that was used to generate at least some of the log data.

17. The method of claim 14 wherein the information caused to be displayed to the user includes the signature.

18. The method of claim 14 wherein the user input received via the second interface includes an instruction from a user to label at least a portion of the signature.

19. The method of claim 14 wherein the user input received via the second interface includes an instruction from the user to treat the signature as important.

20. The method of claim 14 wherein the user input received via the second interface includes an instruction from the user to hide the signature.

21. The method of claim 14 wherein a field included in the signature has a first level of granularity and wherein the set of one or more processors is further configured to, in response to a user request, generate a second signature in which the field has a second level of granularity that is different from the first level of granularity.

22. The method of claim 14 wherein the information caused to be displayed to the user includes a representation of at least a portion of the log data.

23. The method of claim 22 wherein the representation comprises a representation of the cluster.

24. The method of claim 14 wherein the set of one or more processors is further configured to cause a set of controls to be displayed to the user.

25. The method of claim 24 wherein at least one control included in the set of controls comprises a control that allows the user to break a cluster into sub-clusters.

26. The method of claim 24 wherein at least one control included in the set of controls comprises a control that allows the user to combine multiple clusters into a single cluster.

27. A system, comprising:

a first interface configured to receive log data;

a set of one or more processors;

a memory coupled with the set of one or more processors, wherein the memory is configured to provide the set of one or more processors with instructions which when executed cause the set of one or more processors to;

analyze the received log data at least in part by;

clustering portions of the log data into clusters of similar data portions; and

subsequent to the clustering, dynamically generating a signature for each cluster, wherein comparison of subsequent log data with the signature indicates whether the subsequent log data belongs in the cluster; and

cause information associated with the analysis to be displayed to a user;

a storage configured to store the dynamically generated signature, wherein the signature is stored as a result of actions taken by a first user, and wherein the stored signature is modified by a second user; and

a second interface configured to receive user input associated with taking action with respect to the displayed information.

28. The system of claim 27 wherein the information caused to be displayed to the user includes the signature.

29. The system of claim 27 wherein the user input received via the second interface includes an instruction from the user to modify the generated signature.

30. The system of claim 27 wherein the user input received via the second interface includes an instruction from a user to label at least a portion of the signature.

31. The system of claim 27 wherein the user input received via the second interface includes an instruction from the user to treat the signature as important.

32. The system of claim 27 wherein the user input received via the second interface includes an instruction from the user to hide the signature.

33. The system of claim 27 wherein the signature corresponds to a print statement that was used to generate at least some of the log data.

34. The system of claim 27 wherein a field included in the signature has a first level of granularity and wherein the set of one or more processors is further configured to, in response to a user request, generate a second signature in which the field has a second level of granularity that is different from the first level of granularity.

35. The system of claim 27 wherein the information caused to be displayed to the user includes a representation of at least a portion of the log data.

36. The system of claim 35 wherein the representation comprises a representation of the cluster.

37. The system of claim 27 wherein the set of one or more processors is further configured to cause a set of controls to be displayed to the user.

38. The system of claim 37 wherein at least one control included in the set of controls comprises a control that allows the user to break a cluster into sub-clusters.

39. The system of claim 37 wherein at least one control included in the set of controls comprises a control that allows the user to combine multiple clusters into a single cluster.

40. A method, comprising:

receiving, via a first interface, log data;

analyzing, using a set of one or more processors, the received log data at least in part by;

clustering portions of the log data into clusters of similar data portions; and

subsequent to the clustering, dynamically generating a signature for each cluster, wherein comparison of subsequent log data with the signature indicates whether the subsequent log data belongs in the cluster;

causing, using the set of one or more processors, information associated with the analysis to be displayed to a user;

storing the dynamically generated signature, wherein the signature is stored as a result of actions taken by a first user, and wherein the stored signature is modified by a second user; and

receiving, via a second interface, user input associated with taking action with respect to the displayed information.

41. The method of claim 40, wherein the user input received via the second interface includes an instruction from the user to modify the generated signature.

42. The method of claim 40 wherein the signature corresponds to a print statement that was used to generate at least some of the log data.

43. The method of claim 40 wherein the information caused to be displayed to the user includes the signature.

44. The method of claim 40 wherein the user input received via the second interface includes an instruction from a user to label at least a portion of the signature.

45. The method of claim 40 wherein the user input received via the second interface includes an instruction from the user to treat the signature as important.

46. The method of claim 40 wherein the user input received via the second interface includes an instruction from the user to hide the signature.

47. The method of claim 40 wherein a field included in the signature has a first level of granularity and wherein the set of one or more processors is further configured to, in response to a user request, generate a second signature in which the field has a second level of granularity that is different from the first level of granularity.

48. The method of claim 40 wherein the information caused to be displayed to the user includes a representation of at least a portion of the log data.

49. The method of claim 48 wherein the representation comprises a representation of the cluster.

50. The method of claim 40 wherein the set of one or more processors is further configured to cause a set of controls to be displayed to the user.

51. The method of claim 50 wherein at least one control included in the set of controls comprises a control that allows the user to break a cluster into sub-clusters.

52. The method of claim 50 wherein at least one control included in the set of controls comprises a control that allows the user to combine multiple clusters into a single cluster.