Methods and systems of distributed tracing

• US 9,135,145 B2
• Filed: 01/28/2013
• Issued: 09/15/2015
• Est. Priority Date: 01/28/2013
• Status: Expired due to Fees

• Current Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Original Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Inventors: Voccio, Paul, Dietz, Matthew Charles
• Primary Examiner(s): Wong, Don
• Assistant Examiner(s): MORSHED, HOSSAIN M
• Application Number:
  US13/752,147
• Publication Number:
  US 20140215443A1
• Time in Patent Office: 960 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/323 Visualisation of programs or trace data
• CPC Class Codes: G06F 11/3447 Performance evaluation by modeling
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 11/3636 by tracing the execution of the program
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 2201/815 Virtual
• CPC Class Codes: G06F 8/70 Software maintenance or management

Abstract

A system and methods are provided for distributed tracing in a distributed application. In one embodiment, a method includes observing a plurality of messages sent and received among components of the distributed application, generating a probabilistic model of a call flow from observed messages of the distributed system, and constructing a call flow graph based on the probabilistic model for the distributed application. Distributed tracing may include observing messages by performing the subscription-based observation techniques and operations to receive, message traces describing messages being communicated among components of the distributed application. In this regard, the tracing service may merge message traces from different instrumentation points with message traces obtained by observing message queues to generate a probabilistic model and call flow graph. A distributed application developer or other relevant user may benefit in efficiently and effectively tracking down a cause of an abnormal behavior or analyzing performance profiles.

Claims

1. A method of distributed tracing in a distributed application, the method comprising:

observing a plurality of messages sent and received among components of the distributed application, wherein the components of the distributed application communicate with each other through messages, and wherein one or more of the plurality of messages is a request and response message pair;

matching one or more request messages of the plurality of messages with one or more corresponding response messages of the plurality of messages;

generating a probabilistic model of a call flow based on observing the plurality of messages; and

constructing a call flow graph based on the probabilistic model for the distributed application, the call flow graph including a plurality of candidate call flow graphs that capture different orders of response and/or request message arrivals.

2. The method of claim 1, wherein the observing includes receiving a sequence of message traces transmitted by one or more of components of the distributed system, the components modified to transmit message traces in response to sending or receiving of messages.

3. The method of claim 1, wherein the observing includes subscribing to one or more message queues of a messaging service associated with the distributed application.

4. The method of claim 3, wherein the one or more message queues are associated with a message domain limiting message delivery to a portion of the distributed application.

5. The method of claim 1, wherein the observing includes a separate tracing service within each service of the distributed application, the separate tracing service with a separate message domain.

6. The method of claim 1, wherein messages in the distributed system are sent based on a routing envelope and observed by characterizing responses and requests.

7. The method of claim 1, wherein the probabilistic model is a Markov chain.

8. The method of claim 1, wherein the probabilistic model is a hidden Markov model (HMM).

9. The method of claim 1, wherein the call flow graph provides a representation of observed calls in the distributed application as a call stack.

10. The method of claim 1, wherein the call flow graph provides the most probabilistic call stack.

11. The method of claim 1, further comprising debugging the distributed application based at least in part on the call flow graph.

12. The method of claim 1, wherein the plurality of messages are asynchronous messages.

13. The method of claim 1, further comprising:

receiving at least one per-process call stack description from at least one of the components; and

combining the at least one per-process call stack description with the call flow graph to construct a distributed call stack.

14. The method of claim 1, further comprising:

obtaining a sequence of message traces from the observed messages, wherein the message traces identify at least one of a sender of the corresponding message, a receiver of the corresponding message, or whether the corresponding message is a request message or a response message.

15. The method of claim 1, further comprising:

generating profile information associated with the call flow graph.

16. A system comprising:

at least one processor and memory;

a plurality of service components of a distributed application, wherein the plurality of service components is executable on one or more processors, and components of the distributed application communicate with each other through messages;

a message service component to provide communication between the plurality of service components of the distributed application; and

a tracing service component configured to;

observe a plurality of messages sent and received among the components of the distributed application, wherein one or more of the plurality of messages is a request and response message pair,

match one or more request messages of the plurality of messages with one or more corresponding response messages of the plurality of messages,

generate a probabilistic model of a call flow based on observing the plurality of messages, and

construct a call flow graph based on the probabilistic model for the distributed application, the call flow graph including a plurality of candidate call flow graphs that capture different orders of response and/or request message arrivals.

17. The system of claim 16, wherein observing by the tracing service component includes receiving a sequence of message traces transmitted by one or more of components of the distributed system, the components modified to transmit message traces in response to sending or receiving of messages.

18. The system of claim 16, wherein observing by the tracing service component includes subscribing to one or more message queues of the messaging service associated with the distributed application.

19. The system of claim 18, wherein the one or more message queues are associated with a message domain limiting message delivery to a portion of the distributed application.

20. The system of claim 16, wherein observing by the tracing service component includes a separate tracing service within each service of the distributed application, the separate tracing service with a separate message domain.

21. The system of claim 16, wherein messages in the distributed system are sent based on a routing envelope and observed by characterizing responses and requests.

22. The system of claim 16, wherein the probabilistic model is a Markov chain.

23. The system of claim 16, wherein the probabilistic model is a hidden Markov model (HMM).

24. The system of claim 16, wherein the call flow graph provides a representation of observed calls in the distributed application as a call stack.

25. The system of claim 16, wherein the call flow graph provides the most probabilistic call stack.

26. The system of claim 16, further comprising a component to debug the distributed application based at least in part on the call flow graph.

27. The system of claim 16, wherein the plurality of messages are asynchronous messages.

28. The system of claim 16, wherein the tracing service component:

receives at least one per-process call stack description from at least one of the components; and

combines the at least one per-process call stack description with the call flow graph to construct a distributed call stack.

29. The system of claim 16, wherein the tracing service component obtains a sequence of message traces from the observed messages, wherein the message traces identify at least one of a sender of the corresponding message, a receiver of the corresponding message, or whether the corresponding message is a request message or a response message.

30. The system of claim 16, wherein the tracing service component generates profile information associated with the call flow graph.

31. A non-transitory computer-accessible storage medium storing program instructions that when executed by a computer cause the computer to implement a distributed tracing service in a distributed application to:

observe a plurality of messages sent and received among executing components of the distributed application, wherein the components of the distributed application communicate with each other through messages, and wherein one or more of the plurality of messages is a request and response message pair;

match one or more request messages of the plurality of messages with one or more corresponding response messages of the plurality of messages;

generate a probabilistic model of a call flow based on the matching based on observing the plurality of messages; and

construct a call flow graph based on the probabilistic model for the distributed application, the call flow graph including a plurality of candidate call flow graphs that capture different orders of response and/or request message arrivals.