PROGRESSIVE DEPLOYMENT AND TERMINATION OF CANARY INSTANCES FOR SOFTWARE ANALYSIS

• US 20140379901A1
• Filed: 06/25/2013
• Published: 12/25/2014
• Est. Priority Date: 06/25/2013
• Status: Active Grant

• Current Assignee: Netflix, Inc.
• Original Assignee: Netflix, Inc.
• Inventors: Tseitlin, Ariel, Sondow, Joe
• Granted Patent: US 9,225,621 B2
• Time in Patent Office: Days
• US Class Current: 709/224
• CPC Class Codes: G06F 11/368 for test version control, e.g. updating test cases to a new software version
• CPC Class Codes: H04L 41/0896 Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities flow or congestion control using dynamic resource allocation, e.g. in-call renegotiation, H04L47/76
• CPC Class Codes: H04L 41/5025 by proactively reacting to service quality change, e.g. by reconfiguration after service quality degradation or upgrade
• CPC Class Codes: H04L 41/5096 wherein the managed service relates to distributed or central networked applications
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1004 Server selection for load balancing
• CPC Class Codes: H04L 67/1031 Controlling of the operation of servers by a load balancer, e.g. adding or removing servers that serve requests

Abstract

A data processing method, comprising: using computing apparatus, causing instantiating a plurality of baseline application instances that are running a first version of software, and one or more canary application instances that are running a second version of the software; using computing apparatus including a load balancer, causing selectively routing a first proportion of incoming requests to the baseline instances, and routing a second proportion of the incoming requests to the canary instances; monitoring the plurality of canary instances to collect performance data for performance metrics; determining that the performance data indicates a negative performance issue, and in response thereto: using computing apparatus, automatically updating the first proportion to be larger and updating the second proportion to be smaller, and then reconfiguring the load balancer based upon the first proportion and the second proportion; terminating one or more of the canary application instances.

Claims

1. A data processing method, comprising:

using computing apparatus, causing instantiating, a plurality of baseline application instances that are running a first version of software, and one or more canary application instances that are running a second version of the software;

using computing apparatus including a load balancer, causing selectively routing a first proportion of incoming requests to the baseline instances, and routing a second proportion of the incoming requests to the canary instances;

monitoring the plurality of canary instances to collect performance data for a plurality of performance metrics;

determining that the performance data indicates a negative performance issue, and in response thereto;

using computing apparatus, automatically updating the first proportion to be larger and updating the second proportion to be smaller, and then reconfiguring the load balancer based upon the first proportion and the second proportion.

2. The method of claim 1, further comprising:

determining that the performance data does not indicate a negative performance issue, and in response thereto;

using computing apparatus, automatically updating the first proportion to be smaller and updating the second proportion to be larger, and then reconfiguring the load balancer based upon the first proportion and the second proportion.

3. The method of claim 2, further comprising terminating one or more of the baseline application instances

4. The method of claim 1, wherein the determining that the performance data indicates a negative performance issue includes using computing apparatus to evaluate at least one of (i) a threshold amount of variance, (ii) one or more error messages relating to the collected performance data and (iii) a lack of performance data.

5. The method of claim 1, comprising causing instantiating the plurality of baseline application instances in a first cluster and causing instantiating the one or more canary application instances in a second cluster.

6. The method of claim 5, wherein the first cluster and second cluster are defined by metadata that is created and managed using a web application console that is independent of a particular cloud service provider at which the plurality of baseline application instances and the one or more canary application instances have been instantiated.

7. The method of claim 1, comprising causing instantiating the plurality of baseline application instances in a first cluster and causing instantiating the one or more canary application instances in a second cluster, and comprising instantiating the plurality of baseline application instances using a first cloud service provider and causing instantiating the one or more canary application instances using a second cloud service provider that is different from the first cloud service provider.

8. The method of claim 1,

using computing apparatus, a plurality of canary application instances corresponding in number to the baseline application instances;

using computing apparatus including a load balancer, causing selectively routing a first proportion of incoming requests to the baseline instances, and routing a second proportion of the incoming requests to the canary application instances;

using computing apparatus, automatically progressively updating the first proportion to be smaller and updating the second proportion to be larger, reconfiguring the load balancer based upon the first proportion and the second proportion, and progressively terminating the baseline application instances until all the requests are directed only to the canary application instances.

25. The method of claim 1, further comprising terminating one or more of the canary application instances.

9. A non-transitory computer-readable medium storing one or more sequences of instructions which when executed cause performing:

using computing apparatus, causing instantiating, at one or more cloud service providers, a plurality of baseline application instances that are running a first version of software, and one or more canary application instances that are running a second version of the software;

using computing apparatus including a load balancer, causing selectively routing a first proportion of incoming requests to the baseline instances, and routing a second proportion of the incoming requests to the canary instances;

monitoring the plurality of canary instances to collect performance data for a plurality of performance metrics;

determining that the performance data indicates a negative performance issue, and in response thereto;

using computing apparatus, automatically updating the first proportion to be larger and updating the second proportion to be smaller, and then reconfiguring the load balancer based upon the first proportion and the second proportion.

10. The non-transitory computer-readable medium of claim 9, further comprising sequences of instructions which when executed cause:

determining that the performance data does not indicate a negative performance issue, and in response thereto;

using computing apparatus, automatically updating the first proportion to be smaller and updating the second proportion to be larger, and then reconfiguring the load balancer based upon the first proportion and the second proportion.

11. The non-transitory computer-readable medium of claim 10, further comprising sequences of instructions which when executed cause terminating one or more of the baseline application instances

12. The non-transitory computer-readable medium of claim 9, wherein the sequences of instructions which when executed cause determining comprise sequences of instructions which when executed cause determining that the performance data indicates a negative performance issue using computing apparatus to evaluate at least one of (i) a threshold amount of variance, (ii) one or more error messages relating to the collected performance data and (iii) a lack of performance data.

13. The non-transitory computer-readable medium of claim 9, comprising sequences of instructions which when executed cause instantiating the plurality of baseline application instances in a first cluster and causing instantiating the one or more canary application instances in a second cluster.

14. The non-transitory computer-readable medium of claim 13, wherein the first cluster and second cluster are defined by metadata that is created and managed using a web application console that is independent of a particular cloud service provider at which the plurality of baseline application instances and the one or more canary application instances have been instantiated.

15. The non-transitory computer-readable medium of claim 9, comprising sequences of instructions which when executed cause instantiating the plurality of baseline application instances in a first cluster and causing instantiating the one or more canary application instances in a second cluster, and comprising instantiating the plurality of baseline application instances using a first cloud service provider and causing instantiating the one or more canary application instances using a second cloud service provider that is different from the first cloud service provider.

16. The non-transitory computer-readable medium of claim 9, further comprising sequences of instructions which when executed cause:

using computing apparatus, a plurality of canary application instances corresponding in number to the baseline application instances;

using computing apparatus including a load balancer, causing selectively routing a first proportion of incoming requests to the baseline instances, and routing a second proportion of the incoming requests to the canary application instances;

using computing apparatus, automatically progressively updating the first proportion to be smaller and updating the second proportion to be larger, reconfiguring the load balancer based upon the first proportion and the second proportion, and progressively terminating the baseline application instances until all the requests are directed only to the canary application instances.

17. A data processing system comprising:

one or more processors;

a canary analysis component that is communicatively coupled to the one or more processors and to at least one cloud computing environment, the cloud computing environment comprising a load balancer, the canary analysis component comprising a non-transitory computer-readable medium storing one or more sequences of instructions which when executed cause performing;

using computing apparatus, causing instantiating, at one or more cloud service providers, a plurality of baseline application instances that are running a first version of software, and one or more canary application instances that are running a second version of the software;

using computing apparatus including the load balancer, causing selectively routing a first proportion of incoming requests to the baseline instances, and routing a second proportion of the incoming requests to the canary instances;

monitoring the plurality of canary instances to collect performance data for a plurality of performance metrics;

determining that the performance data indicates a negative performance issue, and in response thereto;

using computing apparatus, automatically updating the first proportion to be larger and updating the second proportion to be smaller, and then reconfiguring the load balancer based upon the first proportion and the second proportion;

terminating one or more of the canary application instances.

18. The data processing system of claim 17, the canary analysis component further comprising sequences of instructions which when executed cause:

determining that the performance data does not indicate a negative performance issue, and in response thereto;

using computing apparatus, automatically updating the first proportion to be smaller and updating the second proportion to be larger, and then reconfiguring the load balancer based upon the first proportion and the second proportion.

19. The data processing system of claim 18, the canary analysis component further comprising sequences of instructions which when executed cause terminating one or more of the baseline application instances

20. The data processing system of claim 17, the canary analysis component further comprising sequences of instructions which when executed cause determining that the performance data indicates a negative performance issue includes using computing apparatus to evaluate at least one of (i) a threshold amount of variance, (ii) one or more error messages relating to the collected performance data and (iii) a lack of performance data.

21. The data processing system of claim 17, the canary analysis component further comprising sequences of instructions which when executed cause instantiating the plurality of baseline application instances in a first cluster and causing instantiating the one or more canary application instances in a second cluster.

22. The data processing system of claim 21, wherein the first cluster and second cluster are defined by metadata that is created and managed using a web application console that is independent of a particular cloud service provider at which the plurality of baseline application instances and the one or more canary application instances have been instantiated.

23. The data processing system of claim 17, the canary analysis component further comprising sequences of instructions which when executed cause instantiating the plurality of baseline application instances in a first cluster and causing instantiating the one or more canary application instances in a second cluster, and comprising instantiating the plurality of baseline application instances using a first cloud service provider and causing instantiating the one or more canary application instances using a second cloud service provider that is different from the first cloud service provider.

24. The data processing system of claim 17, the canary analysis component further comprising sequences of instructions which when executed cause:

using computing apparatus, a plurality of canary application instances corresponding in number to the baseline application instances;

using computing apparatus including a load balancer, causing selectively routing a first proportion of incoming requests to the baseline instances, and routing a second proportion of the incoming requests to the canary application instances;

using computing apparatus, automatically progressively updating the first proportion to be smaller and updating the second proportion to be larger, reconfiguring the load balancer based upon the first proportion and the second proportion, and progressively terminating the baseline application instances until all the requests are directed only to the canary application instances.