Using canary instances for software analysis
First Claim
1. A method for evaluating a second version of a software application program, comprising:
- selectively routing, via a load balancing algorithm, incoming requests to a plurality of baseline software instances of the software application program executing on a plurality of hardware platforms and a plurality of canary software instances of the software application program also executing on the plurality of hardware platforms, wherein each of the plurality of baseline software instances comprises a first version of the software application program, each of the plurality of canary software instances comprises the second version of the software application program;
collecting, via one or more hardware processors, performance data for a plurality of performance metrics measuring a performance of one or more computing devices related to the plurality of baseline software instances executing on the plurality of hardware platforms;
collecting, via the one or more hardware processors, performance data for the plurality of performance metrics measuring a performance of one or more other computing devices related to the plurality of canary software instances executing on the plurality of hardware platforms;
for each of the plurality of performance metrics;
calculating, via the one or more hardware processors, an aggregate baseline performance metric for the performance metric by averaging the performance data measuring the performance of the one or more computing devices related to the plurality of baseline software instances; and
calculating, via the one or more hardware processors, a performance value for each canary software instance included in the plurality of canary software instances by calculating a difference between the aggregate baseline performance metric for the performance metric and the performance data measuring the performance of the one or more other computing devices related to a corresponding canary software instance; and
calculating, via the one or more hardware processors, a final measure of performance for the second version of the software application program by at least one of;
calculating an average of the performance values calculated for the canary software instances included in the plurality of canary software instances;
calculating one or more variances of the performance data for one or more of the canary software instances included in the plurality of canary software instances relative to one or more of the aggregate baseline performance metrics calculated for the plurality of baseline software instances; and
calculating an average variance of the performance data for the canary software instances included in the plurality of canary software instances relative to the aggregate baseline performance metrics calculated for the plurality of baseline software instances.
1 Assignment
0 Petitions
Accused Products
Abstract
Techniques for evaluating a second version of software. Embodiments selectively route incoming requests to software instances within a plurality of baseline instances and a plurality of canary instances, where the baseline instances run a first software version and the canary instances run the second software version. The software instances are monitored to collect performance data for a plurality of performance metrics. Embodiments calculate aggregate baseline performance metrics, where each of the aggregate baseline performance metrics is calculated based on the collected performance data for the plurality of baseline instances. For each of the performance metrics and canary instances, embodiments calculate a relative performance value that measures the collected performance data for the respective canary instance and for the respective performance metric, relative to the corresponding aggregate baseline performance metric. A final measure of performance is calculated for the second version of software, based on the relative performance values.
15 Citations
14 Claims
-
1. A method for evaluating a second version of a software application program, comprising:
-
selectively routing, via a load balancing algorithm, incoming requests to a plurality of baseline software instances of the software application program executing on a plurality of hardware platforms and a plurality of canary software instances of the software application program also executing on the plurality of hardware platforms, wherein each of the plurality of baseline software instances comprises a first version of the software application program, each of the plurality of canary software instances comprises the second version of the software application program; collecting, via one or more hardware processors, performance data for a plurality of performance metrics measuring a performance of one or more computing devices related to the plurality of baseline software instances executing on the plurality of hardware platforms; collecting, via the one or more hardware processors, performance data for the plurality of performance metrics measuring a performance of one or more other computing devices related to the plurality of canary software instances executing on the plurality of hardware platforms; for each of the plurality of performance metrics; calculating, via the one or more hardware processors, an aggregate baseline performance metric for the performance metric by averaging the performance data measuring the performance of the one or more computing devices related to the plurality of baseline software instances; and calculating, via the one or more hardware processors, a performance value for each canary software instance included in the plurality of canary software instances by calculating a difference between the aggregate baseline performance metric for the performance metric and the performance data measuring the performance of the one or more other computing devices related to a corresponding canary software instance; and calculating, via the one or more hardware processors, a final measure of performance for the second version of the software application program by at least one of; calculating an average of the performance values calculated for the canary software instances included in the plurality of canary software instances; calculating one or more variances of the performance data for one or more of the canary software instances included in the plurality of canary software instances relative to one or more of the aggregate baseline performance metrics calculated for the plurality of baseline software instances; and calculating an average variance of the performance data for the canary software instances included in the plurality of canary software instances relative to the aggregate baseline performance metrics calculated for the plurality of baseline software instances. - View Dependent Claims (2, 3, 4, 5, 6)
-
-
7. A non-transitory computer-readable medium containing a program that, when executed by one or more hardware processors, performs an operation for evaluating a second version of a software application program, the operation comprising:
-
selectively routing, via a load balancing algorithm, incoming requests to a plurality of baseline software instances of the software application program executing on a plurality of hardware platforms and a plurality of canary software instances of the software application program also executing on the plurality of hardware platforms, wherein each of the plurality of baseline software instances comprises a first version of the software application program, each of the plurality of canary software instances comprises the second version of the software application program; collecting, via the one or more hardware processors, performance data for a plurality of performance metrics measuring a performance of one or more computing devices related to the plurality of baseline software instances executing on the plurality of hardware platforms; collecting, via the one or more hardware processors, performance data for the plurality of performance metrics measuring a performance of one or more other computing devices related to the plurality of canary software instances executing on the plurality of hardware platforms; for each of the plurality of performance metrics; calculating, via the one or more hardware processors, an aggregate baseline performance metric for the performance metric by averaging the performance data measuring the performance of the one or more computing devices related to the plurality of baseline software instances; and calculating, via the one or more hardware processors, a performance value for each canary software instance included in the plurality of canary software instances by calculating a difference between the aggregate baseline performance metric for the performance metric and the performance data measuring the performance of the one or more other computing devices related to a corresponding canary software instance; and calculating, via the one or more hardware processors, a final measure of performance for the second version of the software application program by at least one of; calculating an average of the performance values calculated for the canary software instances included in the plurality of canary software instances; calculating one or more variances of the performance data for one or more of the canary software instances included in the plurality of canary software instances relative to one or more of the aggregate baseline performance metrics calculated for the plurality of baseline software instances; and calculating an average variance of the performance data for the canary software instances included in the plurality of canary software instances relative to the aggregate baseline performance metrics calculated for the plurality of baseline software instances. - View Dependent Claims (8, 9, 10)
-
-
11. A canary analysis system, comprising:
-
a memory storing a canary analysis component; and a hardware processor coupled to the memory, wherein, when executed by the hardware processor, the canary analysis component configures the hardware processor to; selectively route, via a load balancing algorithm, incoming requests to a plurality of baseline software instances of a software application program executing on a plurality of hardware platforms and a plurality of canary software instances of the software application program also executing on the plurality of hardware platforms, wherein each of the plurality of baseline software instances comprises a first version of the software application program, each of the plurality of canary software instances comprises a second version of the software application program; collect performance data for a plurality of performance metrics measuring a performance of one or more computing devices related to the plurality of baseline software instances executing on the plurality of hardware platforms; collect performance data for the plurality of performance metrics measuring a performance of one or more other computing devices related to the plurality of canary software instances executing on the plurality of hardware platforms; for each of the plurality of performance metrics; calculate, via the one or more hardware processors, an aggregate baseline performance metric for the performance metric by averaging the performance data measuring the performance of the one or more computing devices related to the plurality of baseline software instances; and calculate, via the one or more hardware processors, a performance value for each canary software instance included in the plurality of canary software instances by calculating a difference between the aggregate baseline performance metric for the performance metric and the performance data measuring the performance of the one or more other computing devices related to a corresponding canary software instance; and calculate, via the one or more hardware processors, a final measure of performance for the second version of the software application program by at least one of; calculating an average of the performance values calculated for the canary software instances included in the plurality of canary software instances; calculating one or more variances of the performance data for one or more of the canary software instances included in the plurality of canary software instances relative to one or more of the aggregate baseline performance metrics calculated for the plurality of baseline software instances; and calculating an average variance of the performance data for the canary software instances included in the plurality of canary software instances relative to the aggregate baseline performance metrics calculated for the plurality of baseline software instances. - View Dependent Claims (12, 13, 14)
-
Specification