USING CANARY INSTANCES FOR SOFTWARE ANALYSIS

• US 20140282422A1
• Filed: 03/12/2013
• Published: 09/18/2014
• Est. Priority Date: 03/12/2013
• Status: Active Grant

• Current Assignee: Netflix, Inc.
• Original Assignee: Netflix, Inc.
• Inventors: Tuffs, Philip Simon, Rapoport, Roy, Tseitlin, Ariel
• Granted Patent: US 10,318,399 B2
• Time in Patent Office: Days
• US Class Current: 717/127
• CPC Class Codes: G06F 11/3428 Benchmarking
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 2101/00 Indexing scheme relating to the type of digital function generated
• CPC Class Codes: G06F 2201/865 Monitoring of software

Abstract

Techniques for evaluating a second version of software. Embodiments selectively route incoming requests to software instances within a plurality of baseline instances and a plurality of canary instances, where the baseline instances run a first software version and the canary instances run the second software version. The software instances are monitored to collect performance data for a plurality of performance metrics. Embodiments calculate aggregate baseline performance metrics, where each of the aggregate baseline performance metrics is calculated based on the collected performance data for the plurality of baseline instances. For each of the performance metrics and canary instances, embodiments calculate a relative performance value that measures the collected performance data for the respective canary instance and for the respective performance metric, relative to the corresponding aggregate baseline performance metric. A final measure of performance is calculated for the second version of software, based on the relative performance values.

Claims

1. A method for evaluating a second version of software, comprising:

selectively routing incoming requests to software instances within a plurality of software instances, wherein the plurality of software instances comprises a plurality of baseline instances and a plurality of canary instances, wherein the plurality of baseline instances are running a first version of the software, and the plurality of canary instances are running the second version of the software;

monitoring the plurality of software instances to collect performance data for a plurality of performance metrics;

calculating a plurality of aggregate baseline performance metrics, wherein each of the plurality of aggregate baseline performance metrics corresponds to one of the plurality of performance metrics, and wherein each of the plurality of aggregate baseline performance metrics is calculated based on the collected performance data for the plurality of baseline instances, relating to a corresponding one of the plurality of performance metrics;

for each of the plurality of performance metrics and for each of the plurality of canary instances, calculating a relative performance value that measures the collected performance data for the respective canary instance and for the respective performance metric, relative to the corresponding aggregate baseline performance metric from the plurality of aggregate baseline performance metrics; and

calculating a final overall measure of performance for the second version of software, based on the relative performance values.

2. The method of claim 1, wherein the incoming requests are selectively routed to the plurality of software instances according to a load balancing algorithm.

3. The method of claim 1, further comprising:

determining that a portion of performance data relating to one of the plurality of performance metrics for one of the plurality of canary instances fails to satisfy one or more quality criteria; and

excluding the portion of performance data when calculating the final measure of performance for the second version of software, whereby the portion of performance data is not considered in calculating the final measure of performance.

4. The method of claim 3, wherein the one or more quality criteria include at least one of (i) a threshold amount of variance, (ii) one or more error messages relating to the collected performance data and (iii) a lack of performance data.

5. The method of claim 1, wherein monitoring the plurality of software instances to collect performance data further comprises:

deploying a respective monitoring agent for each of the plurality of software instances, wherein each of the monitoring agents is configured to collect instance performance data for the respective software instance and relating to the plurality of performance metrics; and

receiving the collected instance performance data from at least one of the monitoring agents.

6. The method of claim 1, wherein calculating the final measure of performance for the second version of software, based on the relative performance values, further comprises:

for each of the plurality of performance metrics, calculating an aggregate relative canary performance value based on the relative performance values for the plurality of canary instances and for the respective performance metric; and

calculating the final measure of performance for the second version of software, based on the aggregate relative canary performance values for each of the plurality of performance metrics.

7. The method of claim 6, wherein the aggregate relative canary performance values are calculated based on an average of the relative performance values for the plurality of canary instances and for the respective performance metric.

8. The method of claim 6, wherein the final measure of performance for the second version of software is calculated based on an average of the aggregate relative canary performance values for each of the plurality of performance metrics.

9. A non-transitory computer-readable medium containing a program that, when executed, performs an operation for evaluating a second version of software, the operation comprising:

selectively routing incoming requests to software instances within a plurality of software instances, wherein the plurality of software instances comprises a plurality of baseline instances and a plurality of canary instances, wherein the plurality of baseline instances are running a first version of the software, and the plurality of canary instances are running the second version of the software;

monitoring the plurality of software instances to collect performance data for a plurality of performance metrics;

calculating a plurality of aggregate baseline performance metrics, wherein each of the plurality of aggregate baseline performance metrics corresponds to one of the plurality of performance metrics, and wherein each of the plurality of aggregate baseline performance metrics is calculated based on the collected performance data for the plurality of baseline instances, relating to a corresponding one of the plurality of performance metrics;

for each of the plurality of performance metrics and for each of the plurality of canary instances, calculating a relative performance value that measures the collected performance data for the respective canary instance and for the respective performance metric, relative to the corresponding aggregate baseline performance metric from the plurality of aggregate baseline performance metrics; and

calculating a final overall measure of performance for the second version of software, based on the relative performance values.

10. The non-transitory computer-readable medium of claim 9, wherein the incoming requests are selectively routed to the plurality of software instances according to a load balancing algorithm.

11. The non-transitory computer-readable medium of claim 9, the operation further comprising:

determining that a portion of performance data relating to one of the plurality of performance metrics for one of the plurality of canary instances fails to satisfy one or more quality criteria; and

excluding the portion of performance data when calculating the final measure of performance for the second version of software, whereby the portion of performance data is not considered in calculating the final measure of performance.

12. The non-transitory computer-readable medium of claim 11, wherein the one or more quality criteria include at least one of (i) a threshold amount of variance, (ii) one or more error messages relating to the collected performance data and (iii) a lack of performance data.

13. The non-transitory computer-readable medium of claim 9, wherein monitoring the plurality of software instances to collect performance data further comprises:

deploying a respective monitoring agent for each of the plurality of software instances, wherein each of the monitoring agents is configured to collect instance performance data for the respective software instance and relating to the plurality of performance metrics; and

receiving the collected instance performance data from at least one of the monitoring agents.

14. The non-transitory computer-readable medium of claim 9, wherein calculating the final measure of performance for the second version of software, based on the relative performance values, further comprises:

for each of the plurality of performance metrics, calculating an aggregate relative canary performance value based on the relative performance values for the plurality of canary instances and for the respective performance metric; and

calculating the final measure of performance for the second version of software, based on the aggregate relative canary performance values for each of the plurality of performance metrics.

15. The non-transitory computer-readable medium of claim 14, wherein the aggregate relative canary performance values are calculated based on an average of the relative performance values for the plurality of canary instances and for the respective performance metric.

16. The non-transitory computer-readable medium of claim 14, wherein the final measure of performance for the second version of software is calculated based on an average of the aggregate relative canary performance values for each of the plurality of performance metrics.

17. A system, comprising:

a processor; and

a memory containing a program that, when executed by the processor, performs an operation for evaluating a second version of software, comprising;

selectively routing incoming requests to software instances within a plurality of software instances, wherein the plurality of software instances comprises a plurality of baseline instances and a plurality of canary instances, wherein the plurality of baseline instances are running a first version of the software, and the plurality of canary instances are running the second version of the software;

monitoring the plurality of software instances to collect performance data for a plurality of performance metrics;

calculating a plurality of aggregate baseline performance metrics, wherein each of the plurality of aggregate baseline performance metrics corresponds to one of the plurality of performance metrics, and wherein each of the plurality of aggregate baseline performance metrics is calculated based on the collected performance data for the plurality of baseline instances, relating to a corresponding one of the plurality of performance metrics;

for each of the plurality of performance metrics and for each of the plurality of canary instances, calculating a relative performance value that measures the collected performance data for the respective canary instance and for the respective performance metric, relative to the corresponding aggregate baseline performance metric from the plurality of aggregate baseline performance metrics; and

calculating a final overall measure of performance for the second version of software, based on the relative performance values.

18. The system of claim 17, wherein the incoming requests are selectively routed to the plurality of software instances according to a load balancing algorithm.

19. The system of claim 17, the operation further comprising:

determining that a portion of performance data relating to one of the plurality of performance metrics for one of the plurality of canary instances fails to satisfy one or more quality criteria; and

excluding the portion of performance data when calculating the final measure of performance for the second version of software, whereby the portion of performance data is not considered in calculating the final measure of performance.

20. The system of claim 19, wherein the one or more quality criteria include at least one of (i) a threshold amount of variance, (ii) one or more error messages relating to the collected performance data and (iii) a lack of performance data.

21. The system of claim 17, wherein monitoring the plurality of software instances to collect performance data further comprises:

deploying a respective monitoring agent for each of the plurality of software instances, wherein each of the monitoring agents is configured to collect instance performance data for the respective software instance and relating to the plurality of performance metrics; and

receiving the collected instance performance data from at least one of the monitoring agents.

22. The system of claim 17, wherein calculating the final measure of performance for the second version of software, based on the relative performance values, further comprises:

for each of the plurality of performance metrics, calculating an aggregate relative canary performance value based on the relative performance values for the plurality of canary instances and for the respective performance metric; and

calculating the final measure of performance for the second version of software, based on the aggregate relative canary performance values for each of the plurality of performance metrics.

23. The system of claim 22, wherein the aggregate relative canary performance values are calculated based on an average of the relative performance values for the plurality of canary instances and for the respective performance metric.

24. The system of claim 22, wherein the final measure of performance for the second version of software is calculated based on an average of the aggregate relative canary performance values for each of the plurality of performance metrics.