CRITICAL SYSTEMS INSPECTOR

• US 20140281739A1
• Filed: 03/14/2013
• Published: 09/18/2014
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Netflix, Inc.
• Original Assignee: Netflix, Inc.
• Inventors: Rapoport, Roy, Tseitlin, Ariel, Tuffs, Philip Simon
• Granted Patent: US 9,582,395 B2
• Time in Patent Office: Days
• US Class Current: 714/47.2
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3495 for systems

Abstract

Techniques are described for identifying a root cause of a pattern of performance data in a system including a plurality of services. Embodiments provide dependency information for each of the plurality of services, where at least one of the plurality of services is dependent upon a first one of the plurality of services. Each of the plurality of services is monitored to collect performance data for the respective service. Embodiments further analyze the performance data to identify a cluster of services that each follow a pattern of performance data. The first one of the services in the cluster of services is determined to be a root cause of the pattern of performance data, based on the determined dependency information for each of the plurality of services.

Claims

1. A method, comprising:

providing dependency information for each of a plurality of services, wherein at least one of the plurality of services is dependent upon a first one of the plurality of services;

collecting performance data for each respective service of the plurality of services;

analyzing, by operation of one or more computer processors, the performance data to identify a cluster of services that each follow a pattern of performance data; and

determining the first one of the services in the cluster of services is a root cause of the pattern of performance data, based on the determined dependency information for each of the plurality of services.

2. The method of claim 1, wherein analyzing the performance data to identify the cluster of services further comprises determining that the services in the cluster of services have a statistically similar pattern of performance data.

3. The method of claim 1, further comprising:

outputting an interface for display that includes an indication of the first service determined to be the root cause of the pattern of performance data.

4. The method of claim 3, wherein outputting the interface for display, further comprises:

including an indication of a set of related services in the interface, wherein the interface is sorted such that the set of related services are adjacent to the first service determined to be the root cause of the pattern of performance of performance data.

5. The method of claim 1, wherein determining dependency information for each of the plurality of services further comprises at least one of (i) receiving the dependency information from a user and (ii) performing an automated discovery operation to determine the dependency information.

6. The method of claim 1, further comprising:

upon determining the first service is the root cause of the pattern of performance data, receiving a request to determine services within the plurality of services that are being affected by the first service;

analyzing the performance data to identify one or more of the plurality of services that are affected by the first service determined to be the root cause of the pattern of performance data; and

outputting for display an indication of the one or more services.

7. The method of claim 6, wherein analyzing the performance data to identify one or more of the plurality of services that are affected by the first service further comprises:

calculating a similarity value for each of the plurality of services, indicative of a statistical similarity between the respective service and the first service; and

determining that the similarity values calculated for the one or more services exceed a predetermined threshold amount of similarity.

8. The method of claim 1, wherein monitoring each of the plurality of services to collect performance data for the respective service further comprises:

deploying a respective monitoring agent for each of the plurality of services, wherein each of the monitoring agents is configured to monitor a plurality of performance metrics for the respective service, and

wherein determining a first one of the services in the cluster of services is a root cause of the pattern of performance data further comprises;

determining one or more of the plurality of performance metrics, collected for the first service, that is the root cause of the pattern of performance data.

9. A non-transitory computer-readable medium containing a program that, when executed, performs an operation comprising:

providing dependency information for each of a plurality of services, wherein at least one of the plurality of services is dependent upon a first one of the plurality of services;

collecting performance data for each respective service of the plurality of services;

analyzing the performance data to identify a cluster of services that each follow a pattern of performance data; and

determining the first one of the services in the cluster of services is a root cause of the pattern of performance data, based on the determined dependency information for each of the plurality of services.

10. The non-transitory computer-readable medium of claim 9, wherein analyzing the performance data to identify the cluster of services further comprises determining that the services in the cluster of services have a statistically similar pattern of performance data.

11. The non-transitory computer-readable medium of claim 9, the operation further comprising:

outputting an interface for display that includes an indication of the first service determined to be the root cause of the pattern of performance data.

12. The non-transitory computer-readable medium of claim 11, wherein outputting the interface for display, further comprises:

including an indication of a set of related services in the interface, wherein the interface is sorted such that the set of related services are adjacent to the first service determined to be the root cause of the pattern of performance of performance data.

13. The non-transitory computer-readable medium of claim 9, wherein determining dependency information for each of the plurality of services further comprises at least one of (i) receiving the dependency information from a user and (ii) performing an automated discovery operation to determine the dependency information.

14. The non-transitory computer-readable medium of claim 9, the operation further comprising:

upon determining the first service is the root cause of the pattern of performance data, receiving a request to determine services within the plurality of services that are being affected by the first service;

analyzing the performance data to identify one or more of the plurality of services that are affected by the first service determined to be the root cause of the pattern of performance data; and

outputting for display an indication of the one or more services.

15. The non-transitory computer-readable medium of claim 14, wherein analyzing the performance data to identify one or more of the plurality of services that are affected by the first service further comprises:

calculating a similarity value for each of the plurality of services, indicative of a statistical similarity between the respective service and the first service; and

determining that the similarity values calculated for the one or more services exceed a predetermined threshold amount of similarity.

16. The non-transitory computer-readable medium of claim 9, wherein monitoring each of the plurality of services to collect performance data for the respective service further comprises:

deploying a respective monitoring agent for each of the plurality of services, wherein each of the monitoring agents is configured to monitor a plurality of performance metrics for the respective service, and

wherein determining a first one of the services in the cluster of services is a root cause of the pattern of performance data further comprises;

determining one or more of the plurality of performance metrics, collected for the first service, that is the root cause of the pattern of performance data.

17. A system, comprising:

a processor; and

a memory containing a program that, when executed by the processor, performs an operation comprising;

providing dependency information for each of a plurality of services, wherein at least one of the plurality of services is dependent upon a first one of the plurality of services;

collecting performance data for each respective service of the plurality of services;

analyzing the performance data to identify a cluster of services that each follow a pattern of performance data; and

determining the first one of the services in the cluster of services is a root cause of the pattern of performance data, based on the determined dependency information for each of the plurality of services.

18. The system of claim 17, wherein analyzing the performance data to identify the cluster of services further comprises determining that the services in the cluster of services have a statistically similar pattern of performance data.

19. The system of claim 17, the operation further comprising:

outputting an interface for display that includes an indication of the first service determined to be the root cause of the pattern of performance data.

20. The system of claim 19, wherein outputting the interface for display, further comprises:

including an indication of a set of related services in the interface, wherein the interface is sorted such that the set of related services are adjacent to the first service determined to be the root cause of the pattern of performance of performance data.

21. The system of claim 17, wherein determining dependency information for each of the plurality of services further comprises at least one of (i) receiving the dependency information from a user and (ii) performing an automated discovery operation to determine the dependency information.

22. The system of claim 17, the operation further comprising:

upon determining the first service is the root cause of the pattern of performance data, receiving a request to determine services within the plurality of services that are being affected by the first service;

analyzing the performance data to identify one or more of the plurality of services that are affected by the first service determined to be the root cause of the pattern of performance data; and

outputting for display an indication of the one or more services.

23. The system of claim 22, wherein analyzing the performance data to identify one or more of the plurality of services that are affected by the first service further comprises:

calculating a similarity value for each of the plurality of services, indicative of a statistical similarity between the respective service and the first service; and

determining that the similarity values calculated for the one or more services exceed a predetermined threshold amount of similarity.

24. The system of claim 17, wherein monitoring each of the plurality of services to collect performance data for the respective service further comprises:

deploying a respective monitoring agent for each of the plurality of services, wherein each of the monitoring agents is configured to monitor a plurality of performance metrics for the respective service, and

wherein determining a first one of the services in the cluster of services is a root cause of the pattern of performance data further comprises;

determining one or more of the plurality of performance metrics, collected for the first service, that is the root cause of the pattern of performance data.