Progressive deployment and termination of canary instances for software analysis

• US 10,355,950 B2
• Filed: 07/14/2017
• Issued: 07/16/2019
• Est. Priority Date: 06/25/2013
• Status: Active Grant

• Current Assignee: Netflix, Inc.
• Original Assignee: Netflix, Inc.
• Inventors: Tseitlin, Ariel, Sondow, Joseph
• Primary Examiner(s): Lim, Krisna
• Application Number:
  US15/650,810
• Publication Number:
  US 20170317902A1
• Time in Patent Office: 732 Days
• Field of Search: 709224
• CPC Class Codes: G06F 11/368 for test version control, e.g. updating test cases to a new software version
• CPC Class Codes: H04L 41/0896 Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities flow or congestion control using dynamic resource allocation, e.g. in-call renegotiation, H04L47/76
• CPC Class Codes: H04L 41/5025 by proactively reacting to service quality change, e.g. by reconfiguration after service quality degradation or upgrade
• CPC Class Codes: H04L 41/5096 wherein the managed service relates to distributed or central networked applications
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1004 Server selection for load balancing
• CPC Class Codes: H04L 67/1031 Controlling of the operation of servers by a load balancer, e.g. adding or removing servers that serve requests

Abstract

A data processing method, comprising: using computing apparatus, causing instantiating a plurality of baseline application instances that are running a first version of software, and one or more canary application instances that are running a second version of the software; using computing apparatus including a load balancer, causing selectively routing a first proportion of incoming requests to the baseline instances, and routing a second proportion of the incoming requests to the canary instances; monitoring the plurality of canary instances to collect performance data for performance metrics; determining that the performance data indicates a negative performance issue, and in response thereto: using computing apparatus, automatically updating the first proportion to be larger and updating the second proportion to be smaller, and then reconfiguring the load balancer based upon the first proportion and the second proportion; terminating one or more of the canary application instances.

Claims

1. A computer-implemented method, comprising:

routing a first proportion of incoming requests to one or more canary application instances associated with a first version of software application code and a second portion of the incoming requests to one or more application instances associated with a second version of the software application code;

collecting performance data for one or more performance metrics; and

modifying the first proportion of incoming requests according to the one or more performance metrics.

2. The computer-implemented method of claim 1, wherein the one or more performance metrics do not indicate any negative performance issues, and modifying the first proportion of incoming requests comprises increasing the number of incoming requests included in the first proportion of incoming requests.

3. The computer-implemented method of claim 1, wherein the one or more performance metrics do indicate a negative performance issue, and modifying the first proportion of incoming requests comprises decreasing the number of incoming requests included in the first proportion of incoming requests.

4. The computer-implemented method of claim 1, further comprising determining whether the one or more performance metrics indicate a negative performance issue by evaluating at least one of (i) a threshold amount of variance, (ii) one or more error messages associated with the one or more performance metrics, and (iii) how much data has been collected for the one or more performance metrics.

5. The computer-implemented method of claim 1, wherein the one or more application instances associated with the second version of the software application code are instantiated in a first cluster, and the one or more canary application instances are instantiated in a second cluster.

6. The computer-implemented method of claim 5, wherein a first cloud service provider instantiates the one or more application instances associated with the second version of the software application code, and a second cloud service provider different than the first cloud service provider instantiates the one or more canary application instances.

7. A non-transitory computer-readable medium including instructions that, when executed by a processor, cause the processor to perform the steps of:

deploying one or more baseline application instances associated with a first version of software application code and one or more canary application instances associated with a second version of the software application code;

monitoring the one or more canary application instances relative to one or more performance metrics;

collecting performance data for the one or more performance metrics;

determining that the one or more performance metrics do not indicate any negative performance issues; and

deploying one or more additional canary application instances.

8. The non-transitory computer-readable medium of claim 7, further comprising increasing the number of incoming requests included in the first proportion of incoming requests.

9. The non-transitory computer-readable medium of claim 8, wherein deploying one or more additional canary application instances comprises deploying a plurality of additional canary application instances.

10. The non-transitory computer-readable medium of claim 9, further comprising terminating at least one of the baseline application instances included in the one or more baseline application instances.

11. The non-transitory computer-readable medium of claim 10, wherein determining that the one or more performance metrics do not indicate any negative performance issues comprises evaluating at least one of a threshold amount of variance, one or more error messages associated with the one or more performance metrics, and how much data has been collected for the one or more performance metrics.

12. The non-transitory computer-readable medium of claim 11, wherein the one or more baseline application instances are instantiated in a first cluster, and the one or more canary application instances are instantiated in a second cluster.

13. The non-transitory computer-readable medium of claim 12, wherein a first cloud service provider instantiates the one or more baseline application instances, and a second cloud service provider different than the first cloud service provider instantiates the one or more canary application instances.

14. A system, comprising:

a memory that stores a canary analysis engine; and

a processor that is coupled to the memory and, when executing the canary analysis engine, is configured to;

route a first proportion of incoming requests to one or more canary application instances associated with a first version of software application code and a second portion of the incoming requests to one or more application instances associated with a second version of the software application code;

collect performance data for one or more performance metrics;

determine whether the one or more performance metrics indicate a negative performance issue; and

if the one or more performance metrics do not indicate a negative performance issue, cause more incoming requests to be included in the first proportion of incoming requests, or

if the one or more performance metrics indicate a negative performance issue, cause fewer incoming requests to be included in the first proportion of incoming requests.

15. The system of claim 14, wherein the processor is configured to determine whether the one or more performance metrics indicate a negative performance issue by evaluating at least one of (i) a threshold amount of variance, (ii) one or more error messages associated with the one or more performance metrics, and (iii) how much data has been collected for the one or more performance metrics.

16. The system of claim 15, wherein the one or more application instances associated with the second version of the software application code are instantiated in a first cluster, and the one or more canary application instances are instantiated in a second cluster.

17. The system of claim 16, wherein a first cloud service provider instantiates the one or more application instances associated with the second version of the software application code, and a second cloud service provider different than the first cloud service provider instantiates the one or more canary application instances.

18. The system of claim 15, wherein the one or more performance metrics do not indicate a negative performance issue, and the processor is further configured to deploy one or more additional canary application instances and to terminate at least one of the application instances associated with the second version of the software application code.

19. The system of claim 15, wherein the one or more performance metrics indicate a negative performance issue, and the processor is further configured to terminate at least one of the one or more canary application instances.