Log Event Summarization for Distributed Server System

• US 20170220663A1
• Filed: 01/29/2016
• Published: 08/03/2017
• Est. Priority Date: 01/29/2016
• Status: Active Application

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: AppDynamics, Inc. (Cisco Systems, Inc.)
• Inventors: Zhao, Yuchen, Iyer, Arjun
• Granted Patent: US 10,268,750 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/30 Monitoring
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3065 Monitoring arrangements determined by the means or processing involved in reporting the monitored data error or fault reporting or logging G06F11/0766
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/35 Clustering; Classification

Abstract

Clusters of log lines are identified based on log line templates. The log line templates are based on a punctuality pattern for a log line. Clusters of log lines that match each punctuality pattern can be identified based on comparisons between the log lines. The comparison may determine the similarity of the log lines and ultimately identify whether the log lines are close enough to be clustered. The comparison may be based on generated n-grams for the log lines and performing a hash on the n-grams. The resulting cluster information may be communicated to a user in an interface.

Claims

1. A method for monitoring log event data, comprising:

collecting log event data by one or more agents installed on one or more machines that perform a distributed business transaction;

grouping the log event data into groups by data format; and

generating clusters for the log event data within each group.

2. The method of claim 1, wherein the log event data is grouped based on a pattern of fields in the log event data.

3. The method of claim 1, wherein each log event data includes a line of log data, the log lines grouped by a pattern of punctuation in each log line.

4. The method of claim 3, wherein each log line is routed to a particular remote machine based on the grouping of the log line, wherein the particular remote machine is associated with a particular grouping.

5. The method of claim 1, wherein generating clusters includes determining a distance between each log event data within a particular group.

6. The method of claim 5, wherein determining a distance includes generating an n-gram for each log event data.

7. The method of claim 5, wherein determining a distance includes generating a hash for log event data.

8. The method of claim 1, wherein generating clusters includes clustering two log event data together if the distance between the two log event data is within a threshold.

9. The method of claim 1, further including reporting the clusters of log event data from multiple machines in a user interface.

10. A non-transitory computer readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for monitoring log event data, the method comprising:

collecting log event data by one or more agents installed on one or more machines that perform a distributed business transaction;

grouping the log event data into groups by data format;

generating clusters for the log event data within each group; and

reporting the clusters of log event data from multiple machines in a user interface.

11. The non-transitory computer readable storage medium of claim 10, wherein the log event data is grouped based on a pattern of fields in the log event data.

12. The non-transitory computer readable storage medium of claim 10, wherein each log event data includes a line of log data, the log lines grouped by a pattern of punctuation in each log line.

13. The non-transitory computer readable storage medium of claim 12, wherein each log line is routed to a particular remote machine based on the grouping of the log line, wherein the particular remote machine is associated with a particular grouping.

14. The non-transitory computer readable storage medium of claim 10, wherein generating clusters includes determining a distance between each log event data within a particular group.

15. The non-transitory computer readable storage medium of claim 14, wherein determining a distance includes generating an n-gram for each log event data.

16. The non-transitory computer readable storage medium of claim 14, wherein determining a distance includes generating a hash for log event data.

17. The non-transitory computer readable storage medium of claim 10, wherein generating clusters includes clustering two log event data together if the distance between the two log event data is within a threshold.

18. The non-transitory computer readable storage medium of claim 10, further including reporting the clusters of log event data from multiple machines in a user interface.

19. A system for monitoring log event data, the system comprising:

a plurality of machines, each machine including a processor and memory,

one or more modules stored on each of the plurality machines, the one or more modules stored in memory and executable by a corresponding processor to receive log event data from a plurality of agents installed on the plurality of machines that perform a distributed business transaction, the log event data received by a particular machine of the plurality of machines grouped by data format, and generate clusters for the log event data within each group.

20. The system of claim 19, wherein the log event data is grouped based on a pattern of fields in the log event data.

21. The system of claim 19, wherein each log event data includes a line of log data, the log lines grouped by a pattern of punctuation in each log line.

22. The system of claim 19, wherein each log line is received by a particular machine based on the grouping of the log line, wherein the particular machine is associated with a particular grouping.

23. The system of claim 19, the one or more modules further executable to determine a distance between each log event data within a particular group.

24. The system of claim 23, wherein determining a distance includes generating an n-gram for each log event data.

25. The system of claim 23, wherein determining a distance includes generating a hash for log event data.

26. The system of claim 19, the one or more modules further executable to reporting the clusters of log event data from multiple machines in a user interface.