SYSTEMS AND PROCESSES FOR COMPUTER LOG ANALYSIS

• US 20160098342A1
• Filed: 10/05/2015
• Published: 04/07/2016
• Est. Priority Date: 10/05/2014
• Status: Active Grant

• Current Assignee: Yscope Inc.
• Original Assignee: Yscope Inc.
• Inventors: STUMM, Michael, FAIZANULLAH, Muhammad, LION, David, LUO, Yu, YUAN, Ding, ZHAO, Xu, ZHANG, Yongle
• Granted Patent: US 9,729,671 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/323 Visualisation of programs or trace data
• CPC Class Codes: G06F 11/3447 Performance evaluation by modeling
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3608 using formal methods, e.g. model checking, abstract interpretation theorem proving G06N5/013
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 8/443 Optimisation
• CPC Class Codes: H04L 67/34 involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416

Abstract

Existing program code, which is executable on one or more computers forming part of a distributed computer system, is analyzed. The analysis identifies log output instructions present in the program code. Log output instructions are those statements or other code that generate log messages related to service requests processed by the program code. A log model is generated using the analysis. The log model is representative of causal relationships among service requests defined by the program code. The log model can then be applied to logs containing log messages generated by execution of the program code, during its normal operation, to group log messages for improved analysis, including visualization, of the performance and behaviour of the distributed computer system.

Claims

1. A process for performing computer log analysis, the process comprising:

performing an analysis on existing program code that is executable on one or more computers forming part of a distributed computer system, the analysis identifying log output instructions present in the program code, the log output instructions being configured to generate log messages related to service requests processed by the program code;

generating a log model using the analysis, the log model being representative of causal relationships among service requests defined by the program code;

applying the log model to a plurality of log messages generated by execution of the program code to add log messages of the plurality of log messages into one or more groups.

2. The process of claim 1, wherein generating the log model comprises identifying a log output statement in the program code and parsing the log output statement to determine a format of the log output statement and any variable referenced by the log output statement to generate a signature of the log output statement.

3. The process of claim 2, further comprising using a value of a variable referenced by the log output statement to generate the signature for the log output statement.

4. The process of claim 3, storing in the signature of the log output statement as a regular expression.

5. The process of claim 1, wherein generating the log model comprises analyzing dataflow of variables of the program code and storing in the log model one or more indications of one or more variables not modified by a particular service as a request identifier of the particular service request.

6. The process of claim 5, wherein generating the log model comprises identifying a top-level method in the program code as a first method of any thread dedicated to the processing of a single type of service request and storing an identifier of the top-level method in the log model.

7. The process of claim 6, further comprising determining that the particular service request is processed by the top-level method and storing in the log model the request identifier of the particular service request in association with the identifier of the top-level method.

8. The process of claim 1, wherein generating the log model comprises identifying log output statements in the program code and generating indications of temporal associations or dissociations among the log output statements.

9. The process of claim 1, wherein generating the log model comprises identifying log output statements in the program code, identifying threads that contain the log output statements, determining which pairs of the threads communicate with each other, and storing an indication of any determined communicating pairs of threads in the log model.

10. The process of claim 9, wherein determining which pairs of the threads communicate with each other comprises detecting thread creation.

11. The process of claim 9, wherein determining which pairs of the threads communicate with each other comprises determining whether two instructions reachable from two top-level methods operate on a shared object, wherein each of the two top-level methods is a first method of any thread dedicated to the processing of a single type of service request.

12. The process of claim 11, wherein applying the log model comprises adding a particular log message to a particular group on a condition that the particular log message shares a common top-level method with the group.

13. The process of claim 12, wherein the adding is further based on a condition that one or more request identifiers of the particular log message are not different from one or more request identifiers of the group.

14. The process of claim 13, wherein the one or more request identifiers of the particular log message and the one or more request identifiers of the group are indicative of variables in the program code that are not modified during processing of a service request.

15. The process of claim 12, wherein the adding is further based on a condition that a log point of the particular log message matches a temporal sequence in a control flow of the program code.

16. The process of claim 12, further comprising storing the log model at a logging computer of the distributed computer system, the logging computer performing the applying of the log model.

17. The process of claim 1, comprising performing the applying of the log model at a plurality of nodes of the distributed computer system, the plurality of nodes being connected by a network, at least two nodes of the plurality of nodes combining groups of log messages locally without transmitting the log messages over the network before assigning shuffle keys to the groups of log messages and transmitting the groups of log messages over the network to one or more reducer nodes of the plurality of nodes based on the assigned shuffle keys, the one or more reducer nodes further combining received groups of log messages.

18. A log processing system comprising:

a program code analysis engine configured to perform an analysis on existing program code that is executable on one or more computers forming part of a distributed computer system, the analysis identifying log output instructions present in the program code, the log output instructions being configured to generate log messages related to service requests processed by the program code;

the program code analysis engine further configured to use the analysis to generate a log model of causal relationships among service requests defined by the program code; and

a log processing engine configured to apply the log model to a plurality of log messages generated by execution of the program code to add log messages of the plurality of log messages into one or more groups.

19. The system of claim 18, further comprising a results database configured to store indications of the one or more groups.

20. The system of claim 19, further comprising a visualization engine configured to display data contained in the results database.