Performance testing in a software deployment pipeline

• US 10,289,539 B1
• Filed: 09/18/2013
• Issued: 05/14/2019
• Est. Priority Date: 09/18/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Arguelles, Carlos Alejandro, Carre, Arthur, Chandrasekharapuram, Ramakrishnan Hariharan, Tannous, Ayham, Wodicka, Jason Scott
• Primary Examiner(s): Sough, S.
• Assistant Examiner(s): Wang, Ben C
• Application Number:
  US14/030,715
• Time in Patent Office: 2,064 Days
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

Methods and systems for performance testing in a software deployment pipeline are disclosed. One or more performance tests are automatically performed on a build of a software product in a test environment in response to deploying the build to the test environment. One or more performance metrics are collected based on the performance tests. Based on the performance metrics, the build of the software product is accepted or rejected.

Claims

1. A computer-implemented method, comprising:

deploying, based at least in part on a checkin of a software product, a build of the software product to a test environment prior to the build of the software product being deployed to a production environment;

generating a plurality of test transactions;

automatically performing a plurality of performance tests on the build of the software product in the test environment in response to said deploying the build of the software product to the test environment, wherein the performance tests include;

a performance test selected from a load test category, wherein each of the performance tests in the load test category comprises increasing a rate of transactions to the test environment to which the software product has been deployed until a predetermined latency threshold or an error rate threshold is reached, and wherein the performance tests are performed using at least one of the plurality of test transactions generated; and

another performance test selected from a sanity test category or a latency test category;

collecting a plurality of performance metrics based on the plurality of performance tests, wherein at least one performance test is configured in accordance with one or more respective pre-defined specifications;

determining whether the build of the software product passes or fails one or more of the performance tests based on one or more of the performance metrics; and

deploying the build of the software product to the production environment if the build of the software product passes the plurality of performance tests, and rejecting the build of the software product if the build of the software product fails one or more of the performance tests.

2. The method as recited in claim 1, wherein performing the plurality of performance tests on the build of the software product comprises performing one or more sanity tests based on a typical amount of load, and wherein determining whether the build of the software product passes or fails one or more of the performance tests comprises determining whether an error rate for one or more transaction types exceeds a threshold.

3. The method as recited in claim 1, wherein performing the performance tests on the build of the software product comprises performing one or more latency tests, and wherein determining whether the build of the software product passes or fails one or more of the performance tests comprises determining whether the plurality of performance metrics exceed a latency metric for one or more transaction types.

4. The method as recited in claim 1, wherein performing the plurality of performance tests on the build of the software product comprises performing one or more load tests, and wherein determining whether the build of the software product passes or fails the performance tests comprises determining whether one or more hosts achieve a target transaction frequency or a target number of concurrent connections.

5. The method as recited in claim 1, wherein determining whether the build of the software product passes or fails the performance tests based on the performance metrics comprises comparing one or more of the performance metrics to one or more performance metrics for a prior build of the software product.

6. The method as recited in claim 1, further comprising:

performing one or more additional performance tests on a prior build of the software product, wherein the one or more additional performance tests are performed concurrently with the plurality of performance tests; and

collecting one or more additional performance metrics based on the one or more additional performance tests;

wherein determining whether the build of the software product passes or fails one or more of the performance tests based on one or more of the performance metrics comprises comparing at least one of the performance metrics with the one or more additional performance metrics.

7. A system, comprising:

one or more computers configured to implement a software build approval system, wherein the software build approval system comprises;

a transaction generator module configured to generate a plurality of test transactions;

a test environment deployment module configured to deploy, based at least in part on a checkin of a software product, a build of the software product to a test environment prior to the build of the software product being deployed to a production environment, wherein the test environment comprises one or more hosts configured to execute the build of the software product;

a performance testing module configured to;

apply a plurality of performance tests, using the plurality of test transactions generated by the transaction generator module, to the build of the software product in the test environment, wherein the performance tests include;

a performance test selected from a load test category, wherein for the performance tests in the load test category a rate of transactions to the one or more hosts configured to execute the build of the software product is increased until a predetermined latency threshold or an error rate threshold is reached; and

another performance test selected from a sanity test category or a latency test category;

collect a plurality of performance metrics based on the plurality of performance tests; and

determine whether to accept or reject the build of the software product based on the plurality of performance metrics; and

a production environment deployment module configured to;

deploy the build of the software product to a production environment in response to a determination to accept the build of the software product based on the plurality of performance metrics collected; and

reject the build of the software product in response to a determination to reject the build of the software product based on the plurality of performance metrics collected.

8. The system as recited in claim 7, wherein the performance testing module is further configured to:

retrieve one or more performance metrics for a prior build of the software product; and

determine whether to accept or reject the build of the software product based on a comparison between at least one of the plurality of performance metrics for the build of the software product and the one or more performance metrics for the prior build of the software product.

9. The system as recited in claim 7, wherein the performance testing module is further configured to:

apply one or more additional performance tests to a prior build of the software product in an additional test environment;

collect one or more additional performance metrics based on the additional performance tests; and

determine whether to accept or reject the build of the software product based on a comparison between the plurality of performance metrics and the one or more additional performance metrics.

10. The system of claim 7, wherein at least one of the performance tests is configured to specify one or more test parameters to simulate a respective production load.

11. A system, comprising:

one or more computing devices having one or more hardware processors and memory and configured to implement one or more services, wherein the one or more services are configured to;

deploy, based at least in part on a checkin of a software product, a build of a software product to a test environment prior to the build of the software product being deployed to a production environment;

generate a plurality of transactions for the test environment;

automatically cause the build of the software product to process at least some of the generated transactions in performance tests of the test environment, wherein the performance tests include;

a performance test selected from a load test category, wherein for the performance tests in the load test category a rate of transactions to the test environment to which the software product has been deployed is increased until a predetermined latency threshold or an error rate threshold is reached, and wherein the test environment comprises one or more test hosts, each test host configured to simulate a respective production condition; and

another performance test selected from a sanity test category or a latency test category;

collect a plurality of performance metrics for the build of the software product based on the plurality of transactions processed in the test environment;

determine whether to pass or fail the build of the software product based on whether the performance metrics collected satisfy one or more predefined criteria;

automatically deploy the build of the software product to a production environment in response to a determination to pass the build of the software product based on the performance metrics collected; and

automatically reject the build of the software product in response to a determination to fail the build of the software product failed based on the performance metrics collected.

12. The system as recited in claim 11, wherein, in automatically causing the build of the software product to process the plurality of transactions in the test environment, the one or more services are configured to implement one or more sanity tests based on a typical amount of load, and wherein at least one of the performance metrics satisfies a corresponding predefined criterion if an error rate for one or more transaction types does not exceed a threshold.

13. The system as recited in claim 11, wherein, in automatically causing the build of the software product to process the plurality of transactions in the test environment, the one or more services are configured to implement one or more latency tests, and wherein at least one of the performance metrics satisfies a corresponding predefined criterion if the at least one of the performance metrics does not exceed a latency metric for one or more transaction types.

14. The system as recited in claim 11, wherein, in automatically causing the build of the software product to process the plurality of transactions in the test environment, the one or more services are configured to implement one or more load tests, and wherein at least one of the performance metrics satisfies the one or more predefined criteria if one or more hosts achieve a target transaction frequency or a target number of concurrent connections.

15. The system as recited in claim 11, wherein the one or more services are configured to:

automatically cause an additional build of the software product to process a plurality of additional transactions in an additional test environment; and

collect one or more additional performance metrics for the additional build of the software product based on the plurality of additional transactions in the additional test environment;

wherein at least one of the performance metrics satisfies the one or more predefined criteria if one of the performance metrics is within a predefined range of a corresponding one of the one or more additional metrics.

16. A non-transitory computer-readable storage medium storing program instructions computer-executable to perform:

deploying, based on a checkin of a set of program code, the set of program code to a test environment distinct from a production environment and comprising one or more test hosts, prior to the set of program code being deployed to the production environment;

generating a plurality of test transactions;

applying a plurality of performance tests, using at least some of the generated test transactions, to the set of program code in the test environment in response to said deploying the set of program code to the test environment, wherein the plurality of performance tests include;

a performance test selected from a load test category, wherein each of the performance tests in the load test category comprises increasing a rate of transactions to the test environment to which the software product has been deployed until a predetermined latency threshold or an error rate threshold is reached; and

another performance test selected from a sanity test category or a latency test category;

collecting a plurality of performance metrics based on the plurality of performance tests, wherein at least one of the performance tests is configured to simulate a respective production environment condition;

determining whether to accept or reject the set of program code based at least in part on the performance metrics; and

deploying the set of program code to a production environment if said determining determines to accept the set of program code based on the performance metrics, and rejecting the set of program code if said determining determines to reject the set of program code based on the performance metrics.

17. The non-transitory computer-readable storage medium as recited in claim 16, wherein applying the performance tests to the set of program code comprises performing one or more sanity tests based on a typical amount of load, and wherein the set of program code is rejected if an error rate for one or more transaction types exceeds a threshold.

18. The non-transitory computer-readable storage medium as recited in claim 16, wherein applying the performance tests to the set of program code comprises performing one or more latency tests in the test environment, and wherein the set of program code is rejected if one of the performance metrics exceed a latency metric for one or more transaction types.

19. The non-transitory computer-readable storage medium as recited in claim 16, wherein applying the performance tests to the set of program code comprises performing one or more load tests in the test environment, and wherein the set of program code is rejected if the one or more hosts fails to achieve a target transaction frequency or a target number of concurrent connections.

20. The non-transitory computer-readable storage medium as recited in claim 16, wherein a particular performance test is configured to generate a synthetic load that is specified by one or more parameters of a performance test description of the particular performance test.