Critical systems inspector

• US 9,582,395 B2
• Filed: 03/14/2013
• Issued: 02/28/2017
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Netflix, Inc.
• Original Assignee: Netflix, Inc.
• Inventors: Tuffs, Philip Simon, Rapoport, Roy, Tseitlin, Ariel
• Primary Examiner(s): GUYTON, PHILIP A
• Application Number:
  US13/826,942
• Publication Number:
  US 20140281739A1
• Time in Patent Office: 1,447 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3495 for systems

Abstract

Techniques are described for identifying a root cause of a pattern of performance data in a system including a plurality of services. Embodiments provide dependency information for each of the plurality of services, where at least one of the plurality of services is dependent upon a first one of the plurality of services. Each of the plurality of services is monitored to collect performance data for the respective service. Embodiments further analyze the performance data to identify a cluster of services that each follow a pattern of performance data. The first one of the services in the cluster of services is determined to be a root cause of the pattern of performance data, based on the determined dependency information for each of the plurality of services.

Claims

1. A method, comprising:

monitoring a plurality of services;

collecting performance data for each service included in the plurality of services;

identifying, by operation of one or more computer processors, a cluster of related services by analyzing the performance data and determining that each service included in the cluster of related services exhibits a statistically similar pattern of performance data with respect to one or more performance metrics, wherein a first service is included in the cluster of related services, by;

calculating a similarity value for at least one service included in the plurality of services that is indicative of a statistical similarity between the performance data of the service and at least one of the performance data of the first service, the performance data of a second service, and a predetermined statistical pattern, and

determining that the similarity values calculated for each service included in the cluster of related services exceed a predetermined threshold amount of similarity; and

determining, by operation of one or more computer processors, that the first service is a root cause of the pattern of performance data for each service included in the cluster of related services by determining that each service included in the cluster of related services, other than the first service, depends, either directly or indirectly, on the first service.

2. The method of claim 1, further comprising outputting an interface for display that includes an indication of the first service determined to be the root cause of the pattern of performance data.

3. The method of claim 2, wherein outputting the interface for display, further comprises including an indication of a set of related services in the interface, wherein the interface is sorted such that the set of related services are adjacent to the first service determined to be the root cause of the pattern of performance of performance data.

4. The method of claim 1, further comprising determining dependency information for each service included in the plurality of services by performing an automated discovery operation to determine the dependency information.

5. The method of claim 1, further comprising:

upon determining the first service is the root cause of the pattern of performance data, receiving a request from a user to determine services within the plurality of services that are being affected by the first service;

analyzing the performance data to identify one or more of the plurality of services that are affected by the first service; and

outputting for display an indication of the one or more services.

6. The method of claim 5, wherein analyzing the performance data to identify one or more of the plurality of services that are affected by the first service further comprises:

calculating a similarity value for each service included in the plurality of services indicative of a statistical similarity between the respective service and the first service; and

determining that the similarity values calculated for the one or more services exceed a predetermined threshold amount of similarity.

7. The method of claim 1, wherein:

monitoring each service included in the plurality of services comprises deploying a respective monitoring agent for each of the plurality of services, wherein each of the monitoring agents is configured to monitor a plurality of performance metrics for the respective service, and

determining that the first service is a root cause of the pattern of performance data comprises determining one or more of the plurality of performance metrics, collected for the first service, that is the root cause of the pattern of performance data.

8. The method of claim 1, wherein determining that the first service is a root cause of the pattern of performance data further comprises determining that each service included in the cluster of related services depends on the first service.

9. A non-transitory computer-readable medium including instructions that, when executed by a processor, cause the processor to perform one or more operations comprising:

monitoring a plurality of services;

collecting performance data for each service included in the plurality of services;

identifying, by operation of one or more computer processors, a cluster of related services by analyzing the performance data and determining that each service included in the cluster of related services exhibits a statistically similar pattern of performance data with respect to one or more performance metrics, wherein a first service is included in the cluster of related services, by;

calculating a similarity value for at least one service included in the plurality of services that is indicative of a statistical similarity between the performance data of the service and at least one of the performance data of the first service, the performance data of a second service, and a predetermined statistical pattern, and

determining that the similarity values calculated for each service included in the cluster of related services exceed a predetermined threshold amount of similarity; and

determining, by operation of one or more computer processors, that the first service is a root cause of the pattern of performance data for each service included in the cluster of related services by determining that each service included in the cluster of related services, other than the first service, depends, either directly or indirectly, on the first service.

10. The non-transitory computer-readable medium of claim 9, the one or more operations further comprising outputting an interface for display that includes an indication of the first service determined to be the root cause of the pattern of performance data.

11. The non-transitory computer-readable medium of claim 10, wherein outputting the interface for display, further comprises including an indication of a set of related services in the interface, wherein the interface is sorted such that the set of related services are adjacent to the first service determined to be the root cause of the pattern of performance of performance data.

12. The non-transitory computer-readable medium of claim 9, the one or more operations further comprising determining dependency information for each service included in the plurality of services by performing an automated discovery operation to determine the dependency information.

13. The non-transitory computer-readable medium of claim 9, the one or more operations further comprising:

upon determining the first service is the root cause of the pattern of performance data, receiving a request from a user to determine services within the plurality of services that are being affected by the first service;

analyzing the performance data to identify one or more of the plurality of services that are affected by the first service; and

outputting for display an indication of the one or more services.

14. The non-transitory computer-readable medium of claim 13, wherein analyzing the performance data to identify one or more of the plurality of services that are affected by the first service further comprises:

calculating a similarity value for each service included in the plurality of services indicative of a statistical similarity between the respective service and the first service; and

determining that the similarity values calculated for the one or more services exceed a predetermined threshold amount of similarity.

15. The non-transitory computer-readable medium of claim 9, wherein:

monitoring each service included in the plurality of services comprises deploying a respective monitoring agent for each of the plurality of services, wherein each of the monitoring agents is configured to monitor a plurality of performance metrics for the respective service, and

determining that the first service is a root cause of the pattern of performance data comprises determining one or more of the plurality of performance metrics, collected for the first service, that is the root cause of the pattern of performance data.

16. A system, comprising:

a processor; and

a memory including instructions that, when executed by the processor, cause the processor to perform one or more operations comprising;

monitoring a plurality of services;

collecting performance data for each service included in the plurality of services;

identifying, by operation of one or more computer processors, a cluster of related services by analyzing the performance data and determining that each service included in the cluster of related services exhibits a statistically similar pattern of performance data with respect to one or more performance metrics, wherein a first service is included in the cluster of related services, by;

calculating a similarity value for at least one service included in the plurality of services that is indicative of a statistical similarity between the performance data of the service and at least one of the performance data of the first service, the performance data of a second service, and a predetermined statistical pattern, and

determining that the similarity values calculated for each service included in the cluster of related services exceed a predetermined threshold amount of similarity; and

determining that the first service is a root cause of the pattern of performance data for each service included in the cluster of related services by determining that each service included in the cluster of related services, other than the first service, depends, either directly or indirectly, on the first service.

17. The system of claim 16, the operation further comprising outputting an interface for display that includes an indication of the first service determined to be the root cause of the pattern of performance data.

18. The system of claim 17, wherein outputting the interface for display, further comprises including an indication of a set of related services in the interface, wherein the interface is sorted such that the set of related services are adjacent to the first service determined to be the root cause of the pattern of performance of performance data.

19. The system of claim 16, the one or more operations further comprising determining dependency information for each service included in the plurality of services by performing an automated discovery operation to determine the dependency information.

20. The system of claim 16, further comprising:

upon determining the first service is the root cause of the pattern of performance data, receiving a request from a user to determine services within the plurality of services that are being affected by the first service;

analyzing the performance data to identify one or more of the plurality of services that are affected by the first service; and

outputting for display an indication of the one or more services.

21. The system of claim 20, wherein analyzing the performance data to identify one or more of the plurality of services that are affected by the first service further comprises:

calculating a similarity value for each service included in the plurality of services indicative of a statistical similarity between the respective service and the first service; and

determining that the similarity values calculated for the one or more services exceed a predetermined threshold amount of similarity.

22. The system of claim 16, wherein:

monitoring each service included in the plurality of services comprises deploying a respective monitoring agent for each of the plurality of services, wherein each of the monitoring agents is configured to monitor a plurality of performance metrics for the respective service, and

determining that the first service is a root cause of the pattern of performance data comprises determining one or more of the plurality of performance metrics, collected for the first service, that is the root cause of the pattern of performance data.

23. A method, comprising:

determining dependency information for each of a plurality of services, wherein at least one of the plurality of services is dependent upon a first one of the plurality of services;

collecting performance data for each respective service of the plurality of services;

analyzing, by operation of one or more computer processors, the performance data to identify a cluster of services that each follow a pattern of performance data;

determining the first one of the services in the cluster of services is a root cause of the pattern of performance data, based on the determined dependency information for each of the plurality of services;

upon determining the first service is the root cause of the pattern of performance data, receiving a request to determine services within the plurality of services that are being affected by the first service; and

analyzing the performance data to identify one or more of the plurality of services that are affected by the first service determined to be the root cause of the pattern of performance data by;

calculating a similarity value for each of the plurality of services, indicative of a statistical similarity between the respective service and the first service, and

determining that the similarity values calculated for the one or more services exceed a predetermined threshold amount of similarity.