Methods and systems of distributed tracing

• US 9,916,232 B2
• Filed: 08/28/2015
• Issued: 03/13/2018
• Est. Priority Date: 01/28/2013
• Status: Active Grant

• Current Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Original Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Inventors: Voccio, Paul, Dietz, Matthew Charles
• Primary Examiner(s): Morshed, Hossain
• Application Number:
  US14/839,533
• Publication Number:
  US 20150370693A1
• Time in Patent Office: 928 Days
• Field of Search: 717124, 717127, 717128, 717131
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/323 Visualisation of programs or trace data
• CPC Class Codes: G06F 11/3447 Performance evaluation by modeling
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 11/3636 by tracing the execution of the program
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 8/70 Software maintenance or management

Abstract

A system and methods are provided for distributed tracing in a distributed application. In one embodiment, a method includes observing a plurality of messages sent and received among components of the distributed application, generating a probabilistic model of a call flow from observed messages of the distributed system, and constructing a call flow graph based on the probabilistic model for the distributed application. Distributed tracing may include observing messages by performing the subscription-based observation techniques and operations to receive, message traces describing messages being communicated among components of the distributed application. In this regard, the tracing service may merge message traces from different instrumentation points with message traces obtained by observing message queues to generate a probabilistic model and call flow graph. A distributed application developer or other relevant user may benefit in efficiently and effectively tracking down a cause of an abnormal behavior or analyzing performance profiles.

Claims

1. A method of distributed tracing in a distributed application, the method comprising:

observing a plurality of messages sent and received among components of the distributed application, wherein the components of the distributed application communicate with each other through messages, and the plurality of messages is based on one or more user requests;

constructing, based on observing a subset of the plurality of messages, a call flow graph that represents causal relationships between processes of the components;

obtaining a description of one or more call stacks from one or more per-process tracers associated with the distributed application;

recording and matching the per-process tracer description with the corresponding observed messages and selecting appropriate ones of the recorded description;

constructing, based on the call flow graph and the selected description, a distributed call stack indicative of an execution flow across processes of the components in the distributed application; and

using the distributed call stack for analyzing and debugging of the distributed application.

2. The method of claim 1, wherein observing the plurality of messages includes observing the plurality of messages at a plurality of tap points.

3. The method of claim 2, the method further comprising:

generating one or more message traces, wherein one or more message traces describes a response message of the plurality of messages;

merging a plurality of message traces from different tap points into a single stream of observed messages.

4. The method of claim 3, the method further comprising:

time stamping each message trace of the plurality of message traces.

5. The method of claim 1, the method further comprising:

generating profile information associated with the call flow graph.

6. The method of claim 5, wherein the profile information includes a number of times a component of the plurality of components is called.

7. The method of claim 5, wherein the profile information includes an average response time of a component of the plurality of components.

8. The method of claim 1, wherein observing the plurality of messages includes observing a frequency of request messages to a component of the plurality of components and measuring durations between request messages to and response messages from the component.

9. The method of claim 1, wherein the plurality of components includes a database and a messaging service.

10. The method of claim 1, wherein the description is transmitted periodically by processes associated with components in the call flow.

11. The method of claim 1, wherein the description is transmitted at user-defined breakpoints.

12. A system, comprising at least one processor and memory;

a plurality of service components of a distributed application, wherein the plurality of service components is executable on one or more processors, and components of the distributed application communicate with each other through messages;

a message service component to provide communication between the plurality of service components of the distributed application; and

a tracing service component configured to;

observe a plurality of messages sent and received among components of the distributed application, wherein the components of the distributed application communicate with each other through messages, wherein one or more of the plurality of messages is a request and response message pair, and the plurality of messages is based on one or more user requests;

construct, based on the observed plurality of messages, a call flow graph that represents causal relationships between processes of the components;

obtain a description of one or more call stacks from one or more per-process tracers associated with the distributed application;

record and match the per-process tracer description with the corresponding observed messages and select appropriate ones of the recorded description;

construct, based on the call flow graph and the selected description, a distributed call stack indicative of an execution flow across processes of the components in the distributed application; and

use the distributed call stack for analyzing and debugging of the distributed application.

13. The system of claim 12, wherein the tracing service component is configured to:

observe the plurality of messages at a plurality of tap points.

14. The system of claim 13, wherein the tracing service component is configured to:

generate one or more message traces, wherein one or more message traces describes a response message of the plurality of messages; and

merge a plurality of message traces from different tap points into a single stream of observed messages.

15. The system of claim 14, wherein the tracing service component is further configured to:

time stamp each message trace of the plurality of message traces.

16. The system of claim 13, wherein a tap point for observation is at a data link layer.

17. The system of claim 12, wherein the tracing service generates profile information associated with the call flow graph.

18. The system of claim 17, wherein the profile information includes a number of times a component of the plurality of components is called.

19. The system of claim 17, wherein the profile information includes an average response time of a component of the plurality of components.