Diagnosing distributed applications using application logs and request processing paths

• US 9,170,873 B2
• Filed: 08/19/2013
• Issued: 10/27/2015
• Est. Priority Date: 11/14/2012
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Tak, Byung Chul, Tang, Chunqiang, Wang, Long
• Primary Examiner(s): Wilson, Yolanda L
• Application Number:
  US13/970,143
• Publication Number:
  US 20140136692A1
• Time in Patent Office: 799 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/263 Generation of test inputs, e.g. test vectors, patterns or sequences ; with adaptation of the tested hardware for testability with external testers
• CPC Class Codes: G06F 11/323 Visualisation of programs or trace data
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites

Abstract

A system for supporting a distributed application includes a plurality of servers, where each of the plurality of servers includes a thread that processes a request received by the distributed application and a monitoring agent that constructs a transaction path for the request and annotates the transaction path in accordance with a writing action to a log of the distributed application to produce an annotated transaction path. In another embodiment, a system for monitoring a distributed application for errors includes a processor and a computer readable storage medium that stores instructions which, when executed, cause the processor to perform operations including constructing a transaction path for each request received by the distributed application, detecting a writing action to a log of the distributed application, and annotating the transaction path in accordance with the writing action to produce an annotated transaction path.

Claims

1. A system for supporting a distributed application, the system comprising:

a plurality of servers, each of the plurality of servers comprising;

a thread that processes a request received by the distributed application; and

a monitoring agent that constructs a transaction path for the request and annotates the transaction path in accordance with a writing action to a log of the distributed application to produce an annotated transaction path.

2. The system of claim 1, further comprising:

a graphical user interface that displays a graphical representation of the annotated transaction path.

3. The system of claim 2, wherein the graphical representation comprises a sequence of system calls made by the distributed application.

4. The system of claim 3, wherein the graphical user interface visually marks on the annotated transaction path:

an error logged by the distributed application; and

a potential root cause of the error.

5. The system of claim 1, wherein the monitoring agent further maintains a mapping that associates, for each of file that has been opened by the distributed application, a file identifier to a file name string.

6. The system of claim 1, wherein the monitoring agent further injects a fault into the transaction path and stores an association between the fault and a responsive writing action.

7. The system of claim 1, wherein the operations further comprise:

injecting a fault into the transaction path, wherein the writing action is a response to the fault; and

storing an association between the fault and the writing action.

8. The system of claim 7, wherein the injecting comprises:

altering a return value of a system call in the transaction path.

9. The system of claim 7, wherein the operations further comprise:

using the association to diagnose an error detected in the distributed application.

10. A system for monitoring a distributed application for errors, the system comprising:

a processor; and

a computer readable storage medium that stores instructions which, when executed, cause the processor to perform operations comprising;

constructing a transaction path for each request received by the distributed application;

detecting a writing action to a log of the distributed application; and

annotating the transaction path in accordance with the writing action to produce an annotated transaction path.

11. The system of claim 10, wherein the operations further comprise:

detecting that the writing action causes an error to be logged; and

identifying a root cause of the error in accordance with the annotated transaction path.

12. The system of claim 11, wherein the detecting comprises:

finding a keyword in data associated with the writing action, where the keyword indicates an error.

13. The system of claim 11, wherein the identifying comprises:

tracing backward in the annotated transaction path from the writing action, at least until an unexpected activity is detected, wherein the unexpected activity is a candidate for the root cause.

14. The system of claim 13, wherein the tracing results in a plurality of candidates for the root cause being identified, and wherein the plurality of candidates is ranked according to a likelihood of being the root cause.

15. The system of claim 10, wherein the transaction path comprises a sequence of system calls made by the distributed application.

16. The system of claim 10, wherein the annotating comprises:

identifying an event in the transaction path that caused the writing action; and

associating the writing action with the event in the transaction path.

17. The system of claim 16, wherein the identifying comprises:

monitoring files that have been opened by the distributed application; and

detecting that a file identifier specified by a parameter of the event matches a file identifier of one of the files that has been opened by the distributed application.

18. The system of claim 17, wherein the monitoring files comprises:

maintaining a mapping that associates, for each of the files that has been opened by the distributed application, a file identifier to a file name string.

19. The system of claim 18, wherein at least a portion of the maintaining is performed dynamically.

20. The system of claim 18, wherein at least a portion of the maintaining is performed using prior knowledge of logs of the distributed application.