Annotations-based generic load generator engine

• US 9,558,465 B1
• Filed: 12/18/2013
• Issued: 01/31/2017
• Est. Priority Date: 11/11/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Jeffery, Daniel Alexander, Arguelles, Carlos Alejandro, Chandrasekharapuram, Ramakrishnan Hariharan
• Primary Examiner(s): Bullock, Jr., Lewis A
• Assistant Examiner(s): Hebert, Theodore
• Application Number:
  US14/133,554
• Time in Patent Office: 1,140 Days
• Field of Search: 717/131
• US Class Current: 1/1
• CPC Class Codes: G06F 11/263 Generation of test inputs, e.g. test vectors, patterns or sequences ; with adaptation of the tested hardware for testability with external testers
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 11/3644 by instrumenting at runtime
• CPC Class Codes: G06Q 10/063 Operations research, analysis or management
• CPC Class Codes: G06Q 10/0639 Performance analysis of employees; Performance analysis of enterprise or organisation operations
• CPC Class Codes: G06Q 20/08 Payment architectures
• CPC Class Codes: G06Q 30/0206 Price or cost determination based on market factors
• CPC Class Codes: H04L 47/70 Admission control; Resource allocation

Abstract

A generic transaction generator framework for testing a network-based production service may work in conjunction with a product-specific transaction creator module that executes transactions on the service. The transaction creator module may include runtime-discoverable information to communicate product specific details to the framework. Runtime-discoverable information may identify initialization methods, terminate methods, transaction types, transaction methods, transaction dependencies as well as testing parameters, such as transaction rate, testing period and a desired distribution of transaction types. The framework may generate and execute various test transactions and collect performance metrics regarding how well the service performed the test transactions.

Claims

1. A system, comprising:

one or more computers configured to implement a load testing system, wherein the load testing system comprises;

a transaction generation framework; and

a transaction creator module;

wherein the transaction creator module is dynamically bound to the transaction generation framework at runtime;

wherein the transaction generation framework discovers, during runtime, source code annotations within the transaction creator module;

wherein the source code annotations comprise information identifying one or more transaction methods of the transaction creator module, wherein individual ones of the one or more transaction methods cause a test transaction to be performed to a network-based production service configured to process requests from clients for the service via a network;

wherein the transaction generation framework generates one or more test transactions to be performed on the service based on load step information, wherein the one or more test transactions simulate client requests to the service;

wherein for individual ones of the one or more test transactions, the transaction generation framework calls one of the one or more transaction methods of the transaction creator module;

wherein in response to the transaction generation framework calling one of the transaction methods, the transaction creator module communicates with the service to perform the respective test transaction;

wherein the transaction generation framework collects one or more performance metrics based on the service performance of the performed test transactions.

2. The system of claim 1, wherein the source code annotations comprise information identifying a transaction type for individual ones of the one or more transaction methods;

wherein the source code annotations further comprise information indicating ratios among the transaction types;

wherein the transaction generation framework generates the one or more test transactions of the transactions types according to the indicated ratios among the transaction types.

3. The system of claim 2, wherein the transaction generation framework discovers, during runtime, additional source code annotations within the transaction creator module, wherein the additional source code annotations comprise additional information indicating different ratios among the transaction types;

wherein the transaction generation framework generates one or more additional test transactions of the transactions types according to the different ratios among the transaction types.

4. The system of claim 1, wherein the source code annotations further comprise information identifying one or more initialization methods, wherein the transaction generation framework executes the one or more initialization methods according to the source code annotations.

5. The system of claim 1, wherein the source code annotations further comprise information specifying a plurality of endpoints;

wherein individual ones of the plurality of endpoints are network endpoints for sending transactions to an instance of the network-based production service;

wherein the transaction generation framework executes an initialization method of the transaction creator module, wherein the initialization method returns information identifying the plurality of endpoints;

wherein the load testing system sends individual ones of the one or more transactions concurrently to individual ones of the plurality of endpoints;

wherein the transaction generation framework collects and compares performance metrics for the endpoints.

6. A method for testing a network-based production service, comprising;

performing, by one or more computers;

discovering, by a transaction generation framework at runtime, runtime-discoverable information within a transaction creator module, wherein the runtime-discoverable information identifies one or more transaction types to send to the network-based production service configured to process requests from clients for the service via a network;

discovering, by the transaction generation framework at runtime, additional runtime-discoverable information within the transaction creator module, wherein the additional runtime-discoverable information identifies one or more transaction methods of the transaction creator module, wherein at least one of the one or more transaction methods is of one of the one or more transaction types;

generating, by the transaction generation framework, one or more test transactions based on load information specifying a particular distribution of transaction types to be generated, wherein at least one generated test transaction is of one of the transactions types, wherein the one or more test transactions simulate client requests to the service;

calling, by the transaction generation framework for at least one generated test transaction, one of the transaction methods corresponding to the transaction type of the generated test transaction; and

in response to the transaction generation framework calling one of the transaction methods, sending a transaction of the corresponding transaction type to the network-based production service.

7. The method of claim 6, wherein the runtime-discoverable information further comprises information identifying:

an initialization method of the transaction creator module; and

a termination method of the transaction creator module;

wherein the method further comprises;

calling, by the transaction generation framework, the initialization method prior to calling any of the transaction methods; and

calling, by the transaction generation framework, the termination method after calling the one or more transaction methods.

8. The method of claim 7, further comprising passing to the initialization method, by the transaction generation framework, one or more parameters received as initialization parameters to the transaction generation framework.

9. The method of claim 6, wherein the runtime-discoverable information further comprises information identifying multiple initialization methods;

wherein the method further comprises;

executing, by the transaction generation framework, the multiple initialization methods according to the runtime-discoverable information.

10. The method of claim 9, wherein the runtime-discoverable information indicates dependency information specifying a particular dependency order among the multiple initialization methods;

wherein the method further comprises;

executing, by the transaction generation framework, the multiple initialization methods in the particular dependency order based on the indicated dependency information.

11. The method of claim 9, further comprising:

executing, by the transaction generation framework, the multiple initialization methods in parallel.

12. The method of claim 6, wherein the runtime-discoverable information comprises information identifying transaction types for individual ones of the one or more transaction methods;

wherein the runtime-discoverable information comprises information indicating ratios among the transaction types;

wherein the method further comprises;

generating, by the transaction generation framework, the one or more test transactions of the transactions types according to the indicated ratios among the transaction types.

13. A non-transitory, computer-readable storage medium storing program instructions that when executed cause one or more computers to perform:

discovering, at runtime, source code annotations identifying one or more transaction types to send to a network-based production service configured to process requests from clients for the service via a network;

generating one or more test transactions based on load information specifying a distribution of the transaction types, wherein at least one generated test transaction is of one of the transactions types, wherein the one or more test transactions simulate client requests to the service; and

sending individual ones of the generated test transactions to the network-based production service.

14. The non-transitory, computer-readable storage medium of claim 13, wherein the program instructions further cause the one or more computers to perform:

discovering, at runtime, additional source code annotations identifying one or more transaction methods, wherein at least one of the one or more transaction methods corresponds one of the one or more transaction types; and

wherein said executing comprises calling, for at least one generated test transaction, one of the transaction methods corresponding to the transaction type of the generated test transaction, wherein at least one called transaction method causes a transaction of the corresponding type to be sent to the network-based production service.

15. The non-transitory, computer-readable storage medium of claim 13, wherein the program instructions further cause the one or more computers to perform:

collecting one or more performance metrics regarding performance of the test transactions sent to the network-based production service.

16. The non-transitory, computer-readable storage medium of claim 15, wherein the performance metrics comprise one or more of:

transaction latency information;

a number of transactions attempted;

a number of failed transactions;

or

a transaction execution rate.

17. The non-transitory, computer-readable storage medium of claim 13, wherein the program instructions further cause the one or more computers to perform:

discovering at runtime, initialization source code annotations wherein the initialization source code annotations identify one or more initialization methods; and

calling, prior to said generating one or more test transactions, the one or more initialization methods identified by the initialization source code annotations.

18. The non-transitory, computer-readable storage medium of claim 13, wherein the program instructions further cause the one or more computers to perform:

discovering, at runtime, termination source code annotations, wherein the termination source code annotations identify one or more termination methods; and

calling, subsequent to said generating one or more test transactions, the one or more termination methods identified by the termination source code annotations.

19. The non-transitory, computer-readable storage medium of claim 18, wherein the termination source code annotations indicate a plurality of termination methods, wherein the program instructions further cause the one or more computers to perform:

calling the plurality of termination methods in parallel such that the plurality of termination methods execute concurrently.

20. The non-transitory, computer-readable storage medium of claim 18, wherein the termination source code annotations further indicate an order in which the one or more termination methods are to be executed, wherein the program instructions further cause the one or more computers to perform:

calling, by the transaction generation framework, the one or more termination methods in the indicated order.