Using canary instances for software analysis

• US 10,318,399 B2
• Filed: 03/12/2013
• Issued: 06/11/2019
• Est. Priority Date: 03/12/2013
• Status: Active Grant

• Current Assignee: Netflix, Inc.
• Original Assignee: Netflix, Inc.
• Inventors: Tuffs, Philip Simon, Rapoport, Roy, Tseitlin, Ariel
• Primary Examiner(s): Vu, Tuan A
• Application Number:
  US13/796,923
• Publication Number:
  US 20140282422A1
• Time in Patent Office: 2,282 Days
• CPC Class Codes: G06F 11/3428 Benchmarking
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 2101/00 Indexing scheme relating to the type of digital function generated
• CPC Class Codes: G06F 2201/865 Monitoring of software

Abstract

Techniques for evaluating a second version of software. Embodiments selectively route incoming requests to software instances within a plurality of baseline instances and a plurality of canary instances, where the baseline instances run a first software version and the canary instances run the second software version. The software instances are monitored to collect performance data for a plurality of performance metrics. Embodiments calculate aggregate baseline performance metrics, where each of the aggregate baseline performance metrics is calculated based on the collected performance data for the plurality of baseline instances. For each of the performance metrics and canary instances, embodiments calculate a relative performance value that measures the collected performance data for the respective canary instance and for the respective performance metric, relative to the corresponding aggregate baseline performance metric. A final measure of performance is calculated for the second version of software, based on the relative performance values.

Claims

1. A method for evaluating a second version of a software application program, comprising:

selectively routing, via a load balancing algorithm, incoming requests to a plurality of baseline software instances of the software application program executing on a plurality of hardware platforms and a plurality of canary software instances of the software application program also executing on the plurality of hardware platforms, wherein each of the plurality of baseline software instances comprises a first version of the software application program, each of the plurality of canary software instances comprises the second version of the software application program;

collecting, via one or more hardware processors, performance data for a plurality of performance metrics measuring a performance of one or more computing devices related to the plurality of baseline software instances executing on the plurality of hardware platforms;

collecting, via the one or more hardware processors, performance data for the plurality of performance metrics measuring a performance of one or more other computing devices related to the plurality of canary software instances executing on the plurality of hardware platforms;

for each of the plurality of performance metrics;

calculating, via the one or more hardware processors, an aggregate baseline performance metric for the performance metric by averaging the performance data measuring the performance of the one or more computing devices related to the plurality of baseline software instances; and

calculating, via the one or more hardware processors, a performance value for each canary software instance included in the plurality of canary software instances by calculating a difference between the aggregate baseline performance metric for the performance metric and the performance data measuring the performance of the one or more other computing devices related to a corresponding canary software instance; and

calculating, via the one or more hardware processors, a final measure of performance for the second version of the software application program by at least one of;

calculating an average of the performance values calculated for the canary software instances included in the plurality of canary software instances;

calculating one or more variances of the performance data for one or more of the canary software instances included in the plurality of canary software instances relative to one or more of the aggregate baseline performance metrics calculated for the plurality of baseline software instances; and

calculating an average variance of the performance data for the canary software instances included in the plurality of canary software instances relative to the aggregate baseline performance metrics calculated for the plurality of baseline software instances.

2. The method of claim 1, further comprising:

determining that a portion of performance data relating to one of the plurality of performance metrics for one of the plurality of canary software instances fails to satisfy one or more quality criteria; and

excluding the portion of performance data when calculating the final measure of performance for the second version of the software application program, whereby the portion of performance data is not considered in calculating the final measure of performance.

3. The method of claim 2, wherein the one or more quality criteria include at least one of (i) a threshold amount of variance, (ii) one or more error messages relating to the collected performance data, and (iii) a lack of performance data.

4. The method of claim 1, wherein monitoring the plurality of baseline software instances and the plurality of canary software instances to collect performance data comprises:

deploying a different monitoring agent for each of the plurality of baseline software instances and each of the plurality of canary software instances, wherein each of the monitoring agents is configured to collect performance data relating to the plurality of performance metrics; and

receiving the performance data collected by at least one of the monitoring agents.

5. The method of claim 1, further comprising:

determining that a first incoming request routed to a first canary software instance included in the plurality of canary software instances is not processed within a predetermined amount of time; and

in response to the determining, rerouting the first incoming request to a first baseline software instance included in the plurality of baseline software instances.

6. The method of claim 1, further comprising:

detecting an error experienced by a first incoming request routed to a first canary software instance included in the plurality of canary software instances; and

in response to the detecting, rerouting the first incoming request to a first baseline software instance included in the plurality of baseline software instances.

7. A non-transitory computer-readable medium containing a program that, when executed by one or more hardware processors, performs an operation for evaluating a second version of a software application program, the operation comprising:

selectively routing, via a load balancing algorithm, incoming requests to a plurality of baseline software instances of the software application program executing on a plurality of hardware platforms and a plurality of canary software instances of the software application program also executing on the plurality of hardware platforms, wherein each of the plurality of baseline software instances comprises a first version of the software application program, each of the plurality of canary software instances comprises the second version of the software application program;

collecting, via the one or more hardware processors, performance data for a plurality of performance metrics measuring a performance of one or more computing devices related to the plurality of baseline software instances executing on the plurality of hardware platforms;

collecting, via the one or more hardware processors, performance data for the plurality of performance metrics measuring a performance of one or more other computing devices related to the plurality of canary software instances executing on the plurality of hardware platforms;

for each of the plurality of performance metrics;

calculating, via the one or more hardware processors, an aggregate baseline performance metric for the performance metric by averaging the performance data measuring the performance of the one or more computing devices related to the plurality of baseline software instances; and

calculating, via the one or more hardware processors, a performance value for each canary software instance included in the plurality of canary software instances by calculating a difference between the aggregate baseline performance metric for the performance metric and the performance data measuring the performance of the one or more other computing devices related to a corresponding canary software instance; and

calculating, via the one or more hardware processors, a final measure of performance for the second version of the software application program by at least one of;

calculating an average of the performance values calculated for the canary software instances included in the plurality of canary software instances;

calculating one or more variances of the performance data for one or more of the canary software instances included in the plurality of canary software instances relative to one or more of the aggregate baseline performance metrics calculated for the plurality of baseline software instances; and

calculating an average variance of the performance data for the canary software instances included in the plurality of canary software instances relative to the aggregate baseline performance metrics calculated for the plurality of baseline software instances.

8. The non-transitory computer-readable medium of claim 7, the operation further comprising:

determining that a portion of performance data relating to one of the plurality of performance metrics for one of the plurality of canary software instances fails to satisfy one or more quality criteria; and

excluding the portion of performance data when calculating the final measure of performance for the second version of the software application program, whereby the portion of performance data is not considered in calculating the final measure of performance.

9. The non-transitory computer-readable medium of claim 8, wherein the one or more quality criteria include at least one of (i) a threshold amount of variance, (ii) one or more error messages relating to the collected performance data, and (iii) a lack of performance data.

10. The non-transitory computer-readable medium of claim 7, wherein monitoring the plurality of baseline software instances and the plurality of canary software instances to collect performance data comprises:

deploying a different monitoring agent for each of the plurality of baseline software instances and each of the plurality of canary software instances, wherein each of the monitoring agents is configured to collect performance data relating to the plurality of performance metrics; and

receiving the performance data collected by at least one of the monitoring agents.

11. A canary analysis system, comprising:

a memory storing a canary analysis component; and

a hardware processor coupled to the memory, wherein, when executed by the hardware processor, the canary analysis component configures the hardware processor to;

selectively route, via a load balancing algorithm, incoming requests to a plurality of baseline software instances of a software application program executing on a plurality of hardware platforms and a plurality of canary software instances of the software application program also executing on the plurality of hardware platforms, wherein each of the plurality of baseline software instances comprises a first version of the software application program, each of the plurality of canary software instances comprises a second version of the software application program;

collect performance data for a plurality of performance metrics measuring a performance of one or more computing devices related to the plurality of baseline software instances executing on the plurality of hardware platforms;

collect performance data for the plurality of performance metrics measuring a performance of one or more other computing devices related to the plurality of canary software instances executing on the plurality of hardware platforms;

for each of the plurality of performance metrics;

calculate, via the one or more hardware processors, an aggregate baseline performance metric for the performance metric by averaging the performance data measuring the performance of the one or more computing devices related to the plurality of baseline software instances; and

calculate, via the one or more hardware processors, a performance value for each canary software instance included in the plurality of canary software instances by calculating a difference between the aggregate baseline performance metric for the performance metric and the performance data measuring the performance of the one or more other computing devices related to a corresponding canary software instance; and

calculate, via the one or more hardware processors, a final measure of performance for the second version of the software application program by at least one of;

calculating an average of the performance values calculated for the canary software instances included in the plurality of canary software instances;

calculating one or more variances of the performance data for one or more of the canary software instances included in the plurality of canary software instances relative to one or more of the aggregate baseline performance metrics calculated for the plurality of baseline software instances; and

calculating an average variance of the performance data for the canary software instances included in the plurality of canary software instances relative to the aggregate baseline performance metrics calculated for the plurality of baseline software instances.

12. The system of claim 11, wherein the canary analysis component further configures the processor to:

determine that a portion of performance data relating to one of the plurality of performance metrics for one of the plurality of canary software instances fails to satisfy one or more quality criteria; and

exclude the portion of performance data when calculating the final measure of performance for the second version of the software application program, whereby the portion of performance data is not considered in calculating the final measure of performance.

13. The system of claim 12, wherein the one or more quality criteria include at least one of (i) a threshold amount of variance, (ii) one or more error messages relating to the collected performance data, and (iii) a lack of performance data.

14. The system of claim 11, wherein monitoring the plurality of baseline software instances and the plurality of canary software instances to collect performance data comprises:

deploying a different monitoring agent for each of the plurality of baseline software instances and each of the plurality of canary software instances, wherein each of the monitoring agents is configured to collect performance data relating to the plurality of performance metrics; and

receiving the performance data collected by at least one of the monitoring agents.