Diagnosing distributed applications using application logs and request processing paths

• US 9,069,668 B2
• Filed: 11/14/2012
• Issued: 06/30/2015
• Est. Priority Date: 11/14/2012
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Tak, Byung Chul, Tang, Chunqiang, Wang, Long
• Primary Examiner(s): Wilson, Yolanda L
• Application Number:
  US13/676,855
• Publication Number:
  US 20140136896A1
• Time in Patent Office: 958 Days
• Field of Search: 714/32, 714/38.1
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/263 Generation of test inputs, e.g. test vectors, patterns or sequences ; with adaptation of the tested hardware for testability with external testers
• CPC Class Codes: G06F 11/323 Visualisation of programs or trace data
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites

Abstract

A method for monitoring a distributed application for errors includes constructing a transaction path for each request received by the distributed application, detecting a writing action to a log of the distributed application, and annotating the transaction path in accordance with the writing action to produce an annotated transaction path. In another embodiment, a method for building a knowledge base for use in diagnosing an error occurring in a distributed application includes executing the distributed application on a replica of a production server that supports the distributed application, constructing a transaction path for a request received by the distributed application, wherein the transaction path traverses the replica, injecting a fault into the transaction path, detecting a writing action to a log of the distributed application, wherein the writing action is a response to the fault, and storing an association between the fault and the writing action.

Claims

1. A method for monitoring a distributed application for errors, the method comprising:

constructing a transaction path for each request received by the distributed application;

detecting a writing action to a log of the distributed application; and

annotating the transaction path in accordance with the writing action to produce an annotated transaction path.

2. The method of claim 1, further comprising:

detecting that the writing action causes an error to be logged; and

identifying a root cause of the error in accordance with the annotated transaction path.

3. The method of claim 2, wherein the detecting comprises:

finding a keyword in data associated with the writing action, where the keyword indicates an error.

4. The method of claim 2, wherein the identifying comprises:

tracing backward in the annotated transaction path from the writing action, at least until an unexpected activity is detected, wherein the unexpected activity is a candidate for the root cause.

5. The method of claim 4, wherein the unexpected activity is a system call return code.

6. The method of claim 4, wherein the unexpected activity is an error text message.

7. The method of claim 4, wherein the unexpected activity is a deviation from a standard behavior.

8. The method of claim 4, wherein the unexpected activity is a path clustering.

9. The method of claim 4, wherein the tracing results in a plurality of candidates for the root cause being identified.

10. The method of claim 9, wherein the plurality of candidates is ranked according to a likelihood of being the root cause.

11. The method of claim 1, wherein the transaction path comprises a sequence of system calls made by the distributed application.

12. The method of claim 1, wherein the annotating comprises:

identifying an event in the transaction path that caused the writing action; and

associating the writing action with the event in the transaction path.

13. The method of claim 12, wherein the identifying comprises:

monitoring files that have been opened by the distributed application; and

detecting that a file identifier specified by a parameter of the event matches a file identifier of one of the files that has been opened by the distributed application.

14. The method of claim 13, wherein the monitoring files comprises:

maintaining a mapping that associates, for each of the files that has been opened by the distributed application, a file identifier to a file name string.

15. The method of claim 14, wherein at least a portion of the maintaining is performed dynamically.

16. The method of claim 14, wherein at least a portion of the maintaining is performed using prior knowledge of logs of the distributed application.

17. The method of claim 1, further comprising:

injecting a fault into the transaction path, wherein the writing action is a response to the fault; and

storing an association between the fault and the writing action.

18. The method of claim 17, wherein the injecting comprises:

altering a return value of a system call in the transaction path.

19. The method of claim 17, further comprising:

using the association to diagnose an error detected in the distributed application.

20. A method for building a knowledge base for use in diagnosing an error occurring in a distributed application, the method comprising:

executing the distributed application on a replica of a production server that supports the distributed application;

constructing a transaction path for a request received by the distributed application, wherein the transaction path traverses the replica;

injecting a fault into the transaction path;

detecting a writing action to a log of the distributed application, wherein the writing action is a response to the fault;

annotating the transaction path in accordance with the writing action to produce an annotated transaction path; and

storing an association between the fault and the writing action.