Automatic capture of diagnostic data based on transaction behavior learning

• US 8,938,533 B1
• Filed: 07/22/2011
• Issued: 01/20/2015
• Est. Priority Date: 09/10/2009
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: AppDynamics, Inc. (Cisco Systems, Inc.)
• Inventors: Bansal, Jyoti, Sunkara, Bhaskar
• Primary Examiner(s): Mejia, Anthony
• Application Number:
  US13/189,360
• Time in Patent Office: 1,278 Days
• Field of Search: 709/200, 709/224, 717/127, 718/101, 719/317
• US Class Current: 709/224
• CPC Class Codes: H04L 43/022 by sampling
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 43/045 for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

The present technology may determine an anomaly in a portion of a distributed business application. Data can automatically be captured and analyzed for the portion of the application associated with the anomaly. By automatically capturing data for just the portion associated with the anomaly, the present technology reduces the resource and time requirements associated with other code-based solutions for monitoring transactions. A distributed business application is monitored to detect an anomaly based the learned behavior of the business application. The behavior may be learned for each of one or more machines which implement a distributed business transaction. The present system may automatically collect diagnostic data for one or more business transactions and/or requests based on learned behavior for the business transaction or request. Data for a distributed business application transaction may be collected by sampling one or more threads assigned to handle portions of the distributed business transaction. Data regarding the distributed transaction may then be reported from agents monitoring portions of the distributed transaction to one or more controllers and assembled by one or more controllers into business transactions.

Claims

1. A method for monitoring a business transaction, comprising:

accessing runtime data associated with a request executing on a server, the runtime data including aggregated data indicating performance of the request and associated with a first thread running on the server and corresponding with the request, the request received and processed by the server;

executing a module by a processor, the module stored in memory and executable to determine an anomaly associated with the request and based on the runtime data; and

sampling a second thread running on the server and associated with a subsequent instance of the request associated with the anomaly, the second thread sampled at a first rate for a first period of time in response to detecting the anomaly to collect thread data; and

sampling the second thread call stack at a second rate for a second period of time, the second rate less than the first rate, second period of time beginning immediately after the end of the first period of time and extending longer than the first period of time;

updating a thread call stack snapshot based on the call stack samples collected for the second thread; and

reporting the sampled thread data.

2. The method of claim 1, the anomaly determined by comparing performance data for the request to a baseline performance.

3. The method of claim 1, the anomaly determined by an agent residing on an application, the application located on a server of one or more servers forming a distributed system for performing the request.

4. The method of claim 1, the anomaly determined by a remote controller in communication with an agent residing on an application, the application located on a server of one or more servers forming a distributed system for performing the request.

5. The method of claim 1, further including initiating a diagnostic session associated with the anomaly, the diagnostic session associated with a unique identifier, the diagnostic data including the unique identifier.

6. The method of claim 1, the diagnostic data including the performance of one or more outgoing calls performed as part of the request.

7. The method of claim 1, further comprising:

detecting an outgoing call made by the request;

modifying the call to include diagnostic session information; and

transmitting the call made by the request with the modified header.

8. The method of claim 1, wherein thread data is determined from sampling a thread call stack associated with the second thread, the performance of the request determined from the thread data.

9. The method of claim 2, wherein the comparison includes determining a standard deviation for the baseline, the anomaly determined from the standard deviation.

11. The method of claim 1, wherein the anomaly is determined by a controller that receives runtime data collected from a plurality of remote agents.

12. The method of claim 1, wherein the anomaly is determined by an agent that collects the data.

13. The method of claim 6, wherein the outgoing call is to a database.

14. The method of claim 6, wherein the outgoing call is to a network service.

15. The method of claim 7, wherein the call is detected by sampling the second thread.

16. The method of claim 7, wherein the call is detected by instrumented byte code.

17. The method of claim 1, further comprising:

detecting an incoming call;

retrieving diagnostic information from the header of the call, the diagnostic information indicating that a second thread handling the call should be monitored.

18. The method of claim 8, wherein the thread data associated with the second thread is reported as a call graph that indicates the state of the thread call stack over time.

19. The method of claim 8, wherein the thread data includes data for methods executed by the sampled second thread.

10. A method for monitoring a business transaction, comprising:

accessing runtime data associated with a plurality of requests associated with a category, the runtime data associated with a first thread running on a server of the plurality of servers and corresponding with the requests;

executing a module by a processor, the module stored in memory and executable to determine an anomaly associated with the plurality of requests based on the runtime data and a business transaction performance baseline;

automatically sampling a second thread associated with a subsequent instance of a request of the plurality of requests based on the anomaly, the second thread sampled at a first rate for a first period of time;

sampling the second thread call stack at a second rate for a second period of time, the second rate less than the first rate, second period of time beginning immediately after the end of the first period of time and extending longer than the first period of time; and

updating a thread call stack snapshot based on the call stack samples collected for the second thread.

20. The method of claim 10, wherein the anomaly is determined as a multiple of the standard deviation from the baseline.

21. The method of claim 10, wherein the category of requests corresponds to a business transaction.

22. The method of claim 10, wherein the business transaction performance baseline is associated with a rate of outliers for the business transaction.

23. The method of claim 22, wherein the baseline is determined for a period of time for the business transaction on a single machine.

24. The method of claim 22, wherein the baseline is learned from previous runtime data collected for the business transaction.

25. A non-transitory computer readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for monitoring a business transaction, the method comprising:

accessing runtime data associated with a request executing on a server, the runtime data including aggregated data indicating performance of the request and associated with a thread running on the server and corresponding with the request, the request received by the server and processed by the server;

executing a module by a processor, the module stored in memory and executable to determine an anomaly associated with the request and based on the runtime data; and

sampling a second thread running on the server and associated with a subsequent instance of the request associated with the anomaly, the second thread sampled at a first rate for a first period of time in response to detecting the anomaly to collect thread data;

sampling the second thread call stack at a second rate for a second period of time, the second rate less than the first rate, second period of time beginning immediately after the end of the first period of time and extending longer than the first period of time;

updating a thread call stack snapshot based on the call stack samples collected for the second thread; and

reporting the sampled thread data.

26. The non-transitory computer readable storage medium of claim 25, the anomaly determined by comparing performance data for the request to a baseline performance.

27. The non-transitory computer readable storage medium of claim 25, the anomaly determined by an agent residing on an application, the application part of a distributed system for performing the request.

28. The non-transitory computer readable storage medium of claim 25, the anomaly determined by a remote controller in communication with an application, the application part of a distributed system for performing the request.

29. The non-transitory computer readable storage medium of claim 25, further including initiating a diagnostic session associated with the anomaly, the diagnostic session associated with a unique identifier, the diagnostic data including the unique identifier.

30. The non-transitory computer readable storage medium of claim 25, the diagnostic data including the performance of one or more calls performed as part of the request.

31. The non-transitory computer readable storage medium of claim 25, further comprising:

sampling the second thread to detect a call made by the request;

modifying the call to include diagnostic session information;

transmitting the call made by the request with the modified header.

32. The non-transitory computer readable storage medium of claim 25, wherein thread data is determined from sampling a thread call stack associated with the second thread, the performance of the request determined from the thread data.

33. A system for monitoring a business transaction, comprising:

a first application server having a first application; and

a first agent stored in memory of the first application server and executable by a processor to sample a second thread to detect a call to a second application, modify the call to the second application with diagnostic information, and update a thread call stack snapshot based on the call stack samples collected for the second thread, the second thread associated with a subsequent instance of a request associated with an anomaly, the second thread sampled in response to detecting the anomaly at a first rate for a first period of time and at a second rate for a second period of time, the second rate less than the first rate, second period of time beginning immediately after the end of the first period of time and extending longer than the first period of time.

34. The system of claim 33, the system including a controller which receives diagnostic data from one or more agents and transmit a diagnostic session global identifier generated based on the diagnostic data to the first agent, the diagnostic information including the diagnostic session global identifier.

35. The system of claim 34, the system including:

a second application server having a second application; and

a second agent stored in memory of the second application server and executable by a processor to receive the call to the second application and access the diagnostic session global identifier.