System and method for automatic root cause detection

• US 10,248,544 B2
• Filed: 08/28/2017
• Issued: 04/02/2019
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Inventors: Reiss, Andreas
• Primary Examiner(s): Chowdhury, Ziaul A
• Application Number:
  US15/688,222
• Publication Number:
  US 20170357569A1
• Time in Patent Office: 582 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 11/3636 by tracing the execution of the program
• CPC Class Codes: G06F 11/3668 Software testing software testing in telephone exchanges H04M3/242, testing of hardware G06F11/22
• CPC Class Codes: G06F 9/44 Arrangements for executing specific programs

Abstract

A system, method, and computer-readable storage medium are provided to automatically detect a root cause for an error that occurred during execution of functionality in a system. When a request is executed, a log associated with the execution of the request may be stored at the storage. The log may include information associated with a component associated with the execution of the request. Logs related to the same type of request may be compared with the executed request. Specifically, information associated with a component of the executed request may be compared with information associated with a component of the identified stored requests. When information associated with a component of the executed request does not match information associated with a component of the identified stored requests, a root cause for an error is associated with the associated component of the executed request.

Claims

1. A method, comprising:

obtaining, with one or more processors, a history of performance of an application, wherein;

the history of performance comprises a plurality of historical transaction records corresponding to a plurality of transactions serviced by the application responsive to respective requests received at an entry point of the application,

respective historical transaction records identify a plurality of components of the application accessed to service a respective transaction among the plurality of transactions,

respective historical transaction records include a plurality of attributes of operation of respective components among the plurality of components of the application accessed to service the respective transaction, and

the attributes include, for at least some components in at least some transactions, a respective response time of the respective component in the respective transaction;

after obtaining the history of performance, receiving, with one or more processors, an error of the application occurring in a first component of the application during an executed transaction that is servicing a request;

obtaining, with one or more processors, an executed transaction record of the executed transaction associated with the error, the executed transaction record identifying a plurality of components of the application accessed to service the executed transaction;

selecting, with one or more processors, a subset of the historical transaction records at least in part by comparing at least part of the historical transaction records to at least part of the executed transaction record and determining that at least some request parameters and that at least some application components match between the executed transaction record and the subset of historical transaction records, the subset of the historical transaction records including a plurality of historical transaction records;

determining, with one or more processors, that a value of an attribute of the executed transaction record is inconsistent with values of the attribute in the selected subset of historical transaction records, the attribute being associated with a second component of the application, wherein determining that the value of the attribute of the executed transaction record is inconsistent with values of the attribute in the selected subset of historical transaction records comprises comparing a response time of the second component during the executed transaction associated with the error to a threshold response time, the response time of the second component being a portion attributable to the second component of a response time of the request serviced by the executed transaction associated with the error; and

in response to the determination, designating, with one or more processors, in memory, the second component as a potentially associated with a root cause of the error.

2. The method of claim 1, wherein comparing the response time of the second component during the executed transaction to the threshold response time comprises:

determining a difference between the response time of the second component during the executed transaction associated with the error and a response time associated with one or more historical transaction records.

3. The method of claim 1, wherein selecting the subset of the historical transaction records comprises:

selecting the subset of historical transaction records based, at least in part, on the subset of the historical transaction records sharing the entry point to the application with the executed transaction associated with the error, wherein the entry point to the application is one entry point to the application among a plurality of different entry points to the application.

4. The method of claim 1, wherein selecting the subset of the historical transaction records comprises:

selecting the subset of the historical transaction records based, at least in part, on a given request parameter of the subset of the historical transaction records matching a request parameter of the executed transaction associated with the error.

5. The method of claim 1, wherein determining that the value of the attribute of the executed transaction record is inconsistent with values of the attribute in the selected subset of historical transaction records comprises:

determining that a sequence of components called in at least some of the subset of the historical transaction records is different from a sequence of components called in the executed transaction associated with the error.

6. The method of claim 1, wherein:

the obtained executed transaction record identifies five components executed by two or more different hosts servicing the request;

at least one of the components is a database; and

at least one of the components provides the entry point of the application and is mapped to a uniform resource identifier.

7. The method of claim 1, wherein:

at least part of the executed transaction record of the executed transaction associated with the error includes a call stack that identifies a sequence of component calls resulting in the error.

8. The method of claim 1, comprising:

causing the second component to be designated a root cause of the error in a user interface.

9. The method of claim 1, comprising:

instrumenting byte code of the application with instrumentation code; and

obtaining at least some information of the historical transaction records from the instrumentation code and an agent executing on at least one computing device executing at least part of the application.

10. The method of claim 1, wherein determining that the value of the attribute of the executed transaction record is inconsistent with values of the attribute in the selected subset of historical transaction records comprises:

comparing an error-yielding transaction trace that spans a plurality of components servicing the executed transaction associated with the error to a plurality of historical transaction traces of a plurality of transactions matched to the error-yielding transaction trace to determine which of the plurality of components behaved differently in the error-yielding transaction trace relative to behavior of the plurality of components in the historical transaction traces.

11. The method of claim 10, wherein:

the error-yielding transaction trace includes a metric indicative of a duration of time attributable to the second component reported by instrumentation of the second component.

12. The method of claim 1, comprising:

steps for determining a root cause of an error.

13. The method of claim 1, comprising:

measuring application performance of the application, wherein at least part of at least some of the historical transaction records are formed by measuring application performance of the application.

14. The method of claim 13, comprising:

executing the application.

15. A tangible, non-transitory, machine-readable medium storing instructions that when executed by one or more processors effectuate operations comprising:

obtaining, with one or more processors, a history of performance of an application, wherein;

the history of performance comprises a plurality of historical transaction records corresponding to a plurality of transactions serviced by the application responsive to respective requests received at an entry point of the application,

respective historical transaction records identify a plurality of components of the application accessed to service a respective transaction among the plurality of transactions, respective historical transaction records include a plurality of attributes of operation of

respective components among the plurality of components of the application accessed to service the respective transaction, and

the attributes include, for at least some components in at least some transactions, a respective response time of the respective component in the respective transaction;

after obtaining the history of performance, receiving, with one or more processors, an error of the application occurring in a first component of the application during an executed transaction that is servicing a request;

obtaining, with one or more processors, an executed transaction record of the executed transaction associated with the error, the executed transaction record identifying a plurality of components of the application accessed to service the executed transaction;

selecting, with one or more processors, a subset of the historical transaction records at least in part by comparing at least part of the historical transaction records to at least part of the executed transaction record and determining that at least some request parameters and that at least some application components match between the executed transaction record and the subset of historical transaction records, the subset of the historical transaction records including a plurality of historical transaction records;

determining, with one or more processors, that a value of an attribute of the executed transaction record is inconsistent with values of the attribute in the selected subset of historical transaction records, the attribute being associated with a second component of the application, wherein determining that the value of the attribute of the executed transaction record is inconsistent with values of the attribute in the selected subset of historical transaction records comprises comparing a response time of the second component during the executed transaction associated with the error to a threshold response time, the response time of the second component being a portion attributable to the second component of a response time of the request serviced by the executed transaction associated with the error; and

in response to the determination, designating, with one or more processors, in memory, the second component as a potentially associated with a root cause of the error.

16. The medium of claim 15, wherein determining that the value of the attribute of the executed transaction record is inconsistent with values of the attribute in the selected subset of historical transaction records comprises:

determining that a sequence of components called in at least some of the subset of the historical transaction records is different from a sequence of components called in the executed transaction associated with the error.

17. The medium of claim 15, comprising:

instrumenting byte code of the application with instrumentation code; and

obtaining at least some information of the historical transaction records from the instrumentation code and an agent executing on at least one computing device executing at least part of the application.

18. The medium of claim 15, wherein determining that the value of the attribute of the executed transaction record is inconsistent with values of the attribute in the selected subset of historical transaction records comprises:

comparing an error-yielding transaction trace that spans a plurality of components servicing the executed transaction associated with the error to a plurality of historical transaction traces of a plurality of transactions matched to the error-yielding transaction trace to determine which of the plurality of components behaved differently in the error-yielding transaction trace relative to behavior of the plurality of components in the historical transaction traces.

19. The medium of claim 18, wherein:

the error-yielding transaction trace includes a metric indicative of a duration of time attributable to the second component reported by instrumentation of the second component.

20. The medium of claim 15, wherein comparing the response time of the second component during the executed transaction to the threshold response time comprises:

determining a difference between the response time of the second component during the executed transaction associated with the error and a response time associated with one or more historical transaction records.

21. The medium of claim 15, wherein selecting the subset of the historical transaction records comprises:

selecting the subset of historical transaction records based, at least in part, on the subset of the historical transaction records sharing the entry point to the application with the executed transaction associated with the error, wherein the entry point to the application is one entry point to the application among a plurality of different entry points to the application.

22. The medium of claim 15, wherein selecting the subset of the historical transaction records comprises:

selecting the subset of the historical transaction records based, at least in part, on a given request parameter of the subset of the historical transaction records matching a request parameter of the executed transaction associated with the error.

23. The medium of claim 15, wherein:

the obtained executed transaction record identifies five components executed by two or more different hosts servicing the request;

at least one of the components is a database; and

at least one of the components provides the entry point of the application and is mapped to a uniform resource identifier.

24. The medium of claim 15, wherein:

at least part of the executed transaction record of the executed transaction associated with the error includes a call stack that identifies a sequence of component calls resulting in the error.

25. The medium of claim 15, the operations comprising:

causing the second component to be designated a root cause of the error in a user interface.

26. The medium of claim 15, the operations comprising:

measuring application performance of the application, wherein at least part of at least some of the historical transaction records are formed by measuring application performance of the application.