Monitoring of distributed applications

• US 8,893,156 B2
• Filed: 03/24/2009
• Issued: 11/18/2014
• Est. Priority Date: 03/24/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Coelho, Alexandre A., Kurapati, Venkata Seshu Kumar, Letca, Ilarie G., Sanghvi, Ashvinkumar J., Mahajan, Dhananjay M., Sudhakar, Rajeev, Ganesan, Anandha K.
• Primary Examiner(s): Ho, Andy
• Assistant Examiner(s): Kraft, Shih-Wei
• Application Number:
  US12/409,547
• Publication Number:
  US 20100251263A1
• Time in Patent Office: 2,065 Days
• Field of Search: None
• US Class Current: 719/318
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0748 in a remote unit communicating with a single-box computer node experiencing an error/fault remote testing G06F11/2294
• CPC Class Codes: G06F 11/0778 Dumping, i.e. gathering error/state information after a fault for later diagnosis
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3404 for parallel or distributed programming
• CPC Class Codes: G06F 11/3433 for load management allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 2201/865 Monitoring of software
• CPC Class Codes: G06F 2201/875 Monitoring of systems including the internet
• CPC Class Codes: G06F 2201/88 Monitoring involving counting
• CPC Class Codes: G06F 2209/542 Intercept
• CPC Class Codes: G06F 9/546 Message passing systems or structures, e.g. queues

Abstract

Methods, systems, and computer-readable media are disclosed for monitoring a distributed application. A particular method identifies a plurality of components of a distributed application. The method also identifies a specific technology associated with a particular component and attaches a technology specific interceptor to the particular component based on the identified specific technology. The method includes intercepting messages that are sent by or received by the particular component using the technology specific interceptor. At least one potential work item is generated based on the intercepted messages. The method includes determining whether to schedule the at least one potential work item for execution based on a predicted impact of the at least one work potential item on performance of the distributed application.

Claims

1. A monitoring method comprising:

identifying, by a monitoring system, a plurality of components of a distributed application including a first component associated with a first technology at a first computing device and a second component associated with a second technology at a second computing device, wherein the first technology is different from the second technology;

attaching, by the monitoring system, a first technology specific interceptor to the first component based on the associated first technology and intercepting messages that are sent by or received by the first component using the first technology specific interceptor;

attaching, by the monitoring system, a second technology specific interceptor to the second component based on the associated second technology and intercepting messages that are sent by or received by the second component using the second technology specific interceptor;

wherein the first technology specific interceptor and the second technology specific interceptor are distinct; and

in response to detecting a particular intercepted message;

generating at least one potential work item based on the particular intercepted message; and

determining whether to schedule the at least one potential work item based on a predicted impact of the at least one potential work item on performance of the distributed application and determining whether to drop the at least one potential work item without scheduling the at least one potential work item in response to determining that the predicted impact of the at least one potential work item on performance of the distributed application is greater than an acceptable performance impact threshold;

wherein the at least one potential work item is scheduled for execution when the predicted impact of the at least one potential work item on performance is below the acceptable performance impact threshold, and wherein the at least one potential work item is not scheduled for execution when the predicted impact of the at least one potential work item is above the acceptable performance impact threshold.

2. The method of claim 1, wherein the predicted impact of the at least one potential work item indicates an effect on processor utilization at the distributed application, message latency at the distributed application, execution queue length at the distributed application, or any combination thereof.

3. The method of claim 1, wherein the determining of whether to schedule the at least one potential work item includes scheduling based on one or more throttling capabilities.

4. The method of claim 1, wherein the at least one potential work item includes instructions to store discovered dependency information related to the first component and the second component, instructions to store monitored activity information related to at least one of the first component or the second component, instructions to report performance counter information related to at least one of the first component or the second component, or any combination thereof.

5. The method of claim 1, comprising allocating an operation call monitor to track an invocation of a particular function at the first component.

6. The method of claim 5, wherein the particular function has a source operation and a target operation and wherein at least one of the source operation or the target operation is associated with the second component, wherein the at least one potential work item indicates a dependency between the source operation and the target operation.

7. The method of claim 6, further comprising transferring the dependency from a dependency cache to a storage location via a storage service in response to determining that the dependency has not previously been stored.

8. The method of claim 1, further comprising allocating an activity monitor to track activity of a particular execution path at the first component, wherein the at least one potential work item includes activity information related to the particular execution path.

9. The method of claim 1, further comprising allocating a counter monitor to track occurrences of a particular event based on the intercepted messages, wherein the at least one potential work item indicates at least one occurrence of the particular event.

10. The method of claim 1, further comprising:

calculating at least one metric related to the distributed application, wherein the at least one metric includes a number of generated exceptions, a number of reported errors, or any combination thereof;

detecting a failure condition of the distributed application; and

identifying the first component as a potential cause of the failure condition based on the at least one metric.

11. The method of claim 1, wherein the monitoring system receives a request for an empty work item from the first or second technology specific interceptor and refuses the request for the empty work item.

12. The method of claim 1, wherein the monitoring system is configured to turn on or turn off dependency discovery at the first or second technology specific interceptor.

13. The method of claim 1, wherein the particular intercepted message is indicative of a dependency between a source operation at the first component and a target operation at a second component of the plurality of components, and wherein the at least one potential work item includes a work item to store the dependency.

14. The method of claim 1, wherein the particular intercepted message indicates activity information related to an execution path of the distributed application and) the at least one potential work item includes a work item to store the activity information.

15. The method of claim 1, wherein the particular intercepted message indicates a software exception occurrence and the at least one potential work item includes a work item to report the occurrence of the software exception.

16. A monitoring system comprising:

a plurality of technology specific interceptors including a first technology specific interceptor associated with a first technology component of a distributed application located at a first computing device and a second technology specific interceptor associated with a second technology component of the distributed application located at a second computing device, wherein the first technology component is associated with a first technology and the second technology component is associated with a second technology and where the first technology is a different technology from the second technology, the first technology specific interceptor and the second technology specific interceptor are distinct and the first computing device is different from the second computing device, and wherein the monitoring system associates the first technology specific interceptor to the first technology component based on the first technology and associates the second technology specific interceptor to the second technology component based on the second technology; and

a technology agnostic interceptor manager at a third computing device;

wherein the first technology specific interceptor is configured to;

track operation of a first distributed application component of the distributed application, the first distributed application component associated with the first technology, including intercepting messages sent by and received by the first distributed application component;

generate work items responsive to detecting the intercepted messages based on the tracked operation of the first distributed application component; and

send the work items based on the tracked operation of the first distributed application component to the technology agnostic interceptor manager;

wherein the second technology specific interceptor is configured to;

track operation of a second distributed application component of the distributed application, the second distributed application component associated with the second technology, including intercepting messages sent by and received by the second distributed application component;

generate work items based on the tracked operation of the second distributed application component; and

send the work items based on the tracked operation of the second distributed application component to the technology agnostic interceptor manager; and

wherein the technology agnostic interceptor manager includes a plurality of work item managers, including a first work item manager associated with the first technology specific interceptor and a second work item manager associated with the second technology specific interceptor;

wherein each work item manager is associated with a particular technology specific interceptor and is configured to;

receive work items from the particular technology specific interceptor;

queue the received work items in a work item queue;

compare an impact of a particular work item on performance of the distributed application to an acceptable performance impact threshold of the distributed application;

schedule the particular work item for execution when the impact is less than the acceptable performance impact threshold; and

drop the particular work item without scheduling the particular work item for execution when the impact is greater than the acceptable performance impact threshold.

17. The system of claim 16, wherein each technology specific interceptor is further configured to request an empty work item from its associated work item manager, to generate a work item by storing information in the empty work item, and to send the generated work item to the technology agnostic interceptor manager.

18. The system of claim 16, wherein each of the work item managers is further configured to:

limit a number of concurrently executing work items;

limit a size of the work item queue;

reject work items that are received when the work item queue is full;

schedule a particular work item for execution in a foreground thread of a thread pool or in a background thread of the thread pool; and

limit a number of concurrently executing threads of the thread pool.

19. The system of claim 16, wherein the technology agnostic interceptor manager includes one or more monitors allocated by the plurality of technology specific interceptors, wherein the monitors include an identity monitor, an endpoint monitor, a component monitor, an activity monitor, a business perspective monitor, an operation call monitor, a counter monitor, an instrument monitor, or any combination thereof, and wherein the technology agnostic interceptor manager further includes a garbage collector.