AUTOMATIC ROOT CAUSE ANALYSIS OF PERFORMANCE PROBLEMS USING AUTO-BASELINING ON AGGREGATED PERFORMANCE METRICS

• US 20080235365A1
• Filed: 03/20/2007
• Published: 09/25/2008
• Est. Priority Date: 03/20/2007
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: Computer Associates Think Inc. (Broadcom, Inc.)
• Inventors: Seidman, David Isaiah, Bansal, Jyoti Kumar
• Granted Patent: US 7,818,418 B2
• Time in Patent Office: Days
• US Class Current: 709/224
• CPC Class Codes: H04L 43/06 Generation of reports
• CPC Class Codes: H04L 43/0852 Delays
• CPC Class Codes: H04L 43/16 Threshold monitoring

Abstract

Anomalous behavior in a distributed system is automatically detected. Metrics are gathered for transactions, subsystems and/or components of the subsystems. The metrics can identify response times, error counts and/or CPU loads, for instance. Baseline metrics and associated deviation ranges are automatically determined and can be periodically updated. Metrics from specific transactions are compared to the baseline metrics to determine if an anomaly has occurred. A drill down approach can be used so that metrics for a subsystem are not examined unless the metrics for an associated transaction indicate an anomaly. Further, metrics for a component, application which includes one or more components, or process which includes one or more applications, are not examined unless the metrics for an associated subsystem indicate an anomaly. Multiple subsystems can report the metrics to a central manager, which can correlate the metrics to transactions using transaction identifiers or other transaction context data.

Claims

1. A computer-implemented method for detecting anomalous behavior in a distributed system, comprising:

receiving data from a plurality of subsystems in the distributed system which identifies metrics for the plurality of subsystems when the plurality of subsystems perform processing for a plurality of execution paths in the distributed system;

automatically establishing baseline metrics for the subsystems, responsive to the received data;

receiving data from particular subsystems of the plurality of subsystems which identifies metrics for the particular subsystems when the particular subsystems perform processing for a particular execution path in the distributed system;

determining if the metrics for the particular subsystems are anomalous based on the baseline metrics; and

reporting, responsive to the determining.

2. The computer-implemented method of claim 1, wherein:

the metrics for the particular subsystems are determined to be anomalous based on comparisons using the baseline metrics and associated deviation ranges.

3. The computer-implemented method of claim 1, wherein:

the data from the plurality of subsystems is received periodically, and, in response, the baseline metrics are automatically updated periodically.

4. The computer-implemented method of claim 1, further comprising:

automatically establishing deviation ranges associated with the baseline metrics, responsive to the received data from the plurality of subsystems.

5. The computer-implemented method of claim 4, wherein:

the data from the plurality of subsystems is received periodically, and, in response, the deviation ranges are automatically updated periodically.

6. The computer-implemented method of claim 1, wherein:

the reporting comprises identifying one or more of the particular subsystems for which the associated metric is determined to be anomalous.

7. The computer-implemented method of claim 1, wherein:

the plurality of subsystems comprise hosts.

8. The computer-implemented method of claim 1, wherein the data from the plurality of subsystems includes identifiers of the plurality of execution paths.

9. The computer-implemented method of claim 1, wherein the data from the plurality of subsystems includes information identifying calling relationships among the particular subsystems.

10. The computer-implemented method of claim 1, wherein:

the metrics for the plurality of subsystems, the baseline metrics and the metrics for the particular subsystems identify response times.

11. The computer-implemented method of claim 1, wherein:

the metrics for the plurality of subsystems, the baseline metrics and the metrics for the particular subsystems identify error counts.

12. The computer-implemented method of claim 1, wherein:

the metrics for the plurality of subsystems, the baseline metrics and the metrics for the particular subsystems identify CPU loads.

13. The computer-implemented method of claim 1, further comprising:

accessing metrics for components of one or more of the particular subsystems for which the metrics are determined to anomalous;

determining if one or more of the metrics for the components is anomalous; and

reporting, responsive to the determining if the one or more of the metrics for the components is anomalous.

14. The computer-implemented method of claim 13, wherein the accessed metrics include information identifying calling relationships among the components.

15. The computer-implemented method of claim 1, further comprising:

accessing metrics for processes and/or applications of one or more of the particular subsystems for which the metrics are determined to anomalous;

determining if one or more of the metrics for the processes and/or applications is anomalous; and

reporting, responsive to the determining if the one or more of the metrics for the processes and/or applications is anomalous.

16. The computer-implemented method of claim 15, further comprising:

accessing metrics for components of one or more of the processes and/or applications for which the metrics are determined to anomalous;

determining if one or more of the metrics for the components is anomalous; and

reporting, responsive to the determining if the one or more of the metrics for the components is anomalous.

17. A computer-implemented method for detecting anomalous behavior at a host, comprising:

providing metrics for a plurality of components of at least one host, the plurality of components are associated with a plurality of execution paths;

automatically establishing baseline metrics for the plurality of components based on the provided metrics for the plurality of components;

providing metrics for particular components of the at least one host, the particular components are in a particular execution path;

determining if the metrics for the particular components are anomalous based on the baseline metrics; and

reporting, responsive to the determining.

18. The computer-implemented method of claim 17, wherein:

the metrics for the particular components are determined to be anomalous based on comparisons using the baseline metrics and associated deviation ranges.

19. The computer-implemented method of claim 17, wherein:

the metrics for the plurality of components are provided periodically, and, in response, the baseline metrics are automatically updated periodically.

20. The computer-implemented method of claim 17, wherein:

the reporting comprises identifying one or more of the particular components for which the associated metric is determined to be anomalous.

21. The computer-implemented method of claim 17, further comprising:

providing data of the at least one host which identifies the plurality of execution paths.

22. The computer-implemented method of claim 17, further comprising:

providing data of the at least one host which identifies calling relationships among the particular components.

23. The computer-implemented method of claim 17, wherein:

the baseline metrics and the metrics for the particular components identify response times.

24. The computer-implemented method of claim 17, wherein:

the baseline metrics and the metrics for the particular components identify error counts.

25. The computer-implemented method of claim 17, wherein:

the baseline metrics and the metrics for the particular components identify CPU loads.

26. The computer-implemented method of claim 17, further comprising:

automatically establishing deviation ranges associated with the baseline metrics, responsive to the metrics for the plurality of components.

27. The computer-implemented method of claim 26, wherein:

the metrics for the plurality of components are provided periodically, and, in response, the deviation ranges are automatically updated periodically.

28. At least one processor readable storage device having processor readable code embodied thereon for programming at least one processor to perform a method, the method comprising:

receiving data from a plurality of subsystems in the distributed system which identifies metrics for the plurality of subsystems when the plurality of subsystems perform processing for a plurality of execution paths in the distributed system;

automatically establishing baseline metrics for the subsystems, responsive to the received data;

receiving data from particular subsystems of the plurality of subsystems which identifies metrics for the particular subsystems when the particular subsystems perform processing for a particular execution path in the distributed system;

determining if the metrics for the particular subsystems are anomalous based on the baseline metrics; and

reporting, responsive to the determining.

29. The at least one processor readable storage device of claim 28, wherein:

the reporting comprises identifying one or more of the particular subsystems for which the associated metric is determined to be anomalous.

30. The at least one processor readable storage device of claim 28, wherein the data from the plurality of subsystems includes identifiers of the plurality of execution paths.

31. The at least one processor readable storage device of claim 28, wherein the data from the plurality of subsystems includes information identifying calling relationships among the particular subsystems.

32. The at least one processor readable storage device of claim 28, wherein the method performed further comprises:

accessing metrics for components of one or more of the particular subsystems for which the metrics are determined to anomalous;

determining if one or more of the metrics for the components is anomalous; and

reporting, responsive to the determining if the one or more of the metrics for the components is anomalous.

33. At least one processor readable storage device having processor readable code embodied thereon for programming at least one processor to perform a method, the method comprising:

providing metrics for a plurality of components of at least one host, the plurality of components are associated with a plurality of execution paths;

automatically establishing baseline metrics for the plurality of components based on the provided metrics for the plurality of components;

providing metrics for particular components of the at least one host, the particular components are in a particular execution path;

determining if the metrics for the particular components are anomalous based on the baseline metrics; and

reporting, responsive to the determining.

34. The at least one processor readable storage device of claim 33, wherein:

the reporting comprises identifying one or more of the particular components for which the associated metric is determined to be anomalous.

35. The at least one processor readable storage device of claim 33, wherein the method performed further includes providing data of the at least one host which identifies the plurality of execution paths.

36. The at least one processor readable storage device of claim 33, wherein the method performed further includes providing data of the at least one host which identifies calling relationships among the particular components.