Software testing environment that includes a duplicating proxy service

• US 9,836,388 B1
• Filed: 09/26/2013
• Issued: 12/05/2017
• Est. Priority Date: 09/26/2013
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Moniz, Samuel Leonard, Beranek, Kevin Michael, Christopher, Keian, Evenson, Andrew Ross, Kaulia, Christopher Keakini, Peterson, Kyle Bradley
• Primary Examiner(s): Zhen, Li B
• Assistant Examiner(s): Wei, Daniel
• Application Number:
  US14/038,619
• Time in Patent Office: 1,531 Days
• Field of Search: 717124
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 11/3672 Test management
• CPC Class Codes: G06F 11/368 for test version control, e.g. updating test cases to a new software version
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

The techniques described herein provide software testing of a candidate software system. In some examples, an interceptor intercepts at least one production request to a production software system. The duplicating proxy service causes the at least one intercepted request to be processed by the candidate software system being validated and the production software system being used to validate the candidate software system. The duplicating proxy service may then compare and/or analyze at least one candidate response to the intercepted request from the candidate software system and at least one production response to the intercepted request from the production software system.

Claims

1. A duplicating proxy system, comprising:

at least one computing device configured to implement an interceptor service and a duplicating proxy service,

the interceptor service to;

intercept a production request to a production system that is to provide information responsive to the production request;

duplicate, at least partially, the production request to create a duplicate request;

provide the production request to the production system; and

provide the duplicate request to the duplicating proxy service; and

the duplicating proxy service to;

cause a candidate shadow request to be processed by a candidate software system, the candidate shadow request based at least in part on the duplicate request and the candidate software system being a modified version of the production system that is to be validated or a replacement for the production system;

cause an authority shadow request to be processed by an authority software system that is distinct from the production system and the candidate software system, the authority shadow request based at least in part on the duplicate request, wherein the candidate software system is validated by the authority software system based at least in part on a comparison of the candidate software system to the authority software system;

receive, in response to the candidate shadow request, a candidate response from the candidate software system;

receive, in response to the authority shadow request, an authority response from the authority software system; and

compare the candidate response to the authority response to identify differences between the candidate response and the authority response.

2. The duplicating proxy system as recited in claim 1, wherein the duplicating proxy service is further configured to identify a latency difference between the candidate software system and the authority software system based at least in part on the candidate response and the authority response.

3. The duplicating proxy system as recited in claim 1, wherein the duplicating proxy service is further configured to identify at least one difference that is specified as being unacceptable to operation of the candidate software system.

4. The duplicating proxy system as recited in claim 3, wherein the duplicating proxy service is further configured to:

record the authority shadow request, the candidate shadow request, or both;

replay at least one of the authority shadow request or the candidate shadow request to a modified candidate software system that generates a replay candidate response; and

determine that the at least one difference that is specified as being unacceptable to the operation of the candidate software system that occurred in the candidate response occurs in the replay candidate response.

5. The duplicating proxy system as recited in claim 1, wherein a production software system modifies system data of the production system and the duplicating proxy service causes the production request to be processed by the candidate software system such that system data of the production system is not modified.

6. A computer-implemented method comprising:

under control of one or more processors configured with executable instructions,

requesting a candidate software system to process at least one candidate request that is based at least in part on an intercepted request to a production software system from a requester, the candidate software system being a modified version of the production software system that is to be validated or a replacement for the production software system and undergoing validation prior to being placed in production;

requesting an authority software system to process at least one authority request that is based at least in part on the intercepted request, the authority software system being used for validating the candidate software system, wherein the production software system is a live software system that is different than the authority software system and the candidate software system;

receiving, in response to processing of the at least one candidate request, at least one candidate response from the candidate software system;

receiving, in response to processing of the at least one authority request, at least one authority response from the authority software system;

classifying a comparison between the at least one candidate response and the at least one authority response; and

generating a validation result based on the comparison between the at least one candidate response and the at least one authority response.

7. The computer-implemented method as recited in claim 6, further comprising determining a latency difference between the candidate software system and the authority software system based at least in part on the at least one candidate response and the at least one authority response.

8. The computer-implemented method as recited in claim 6, further comprising intercepting the intercepted request.

9. The computer-implemented method as recited in claim 8, further comprising:

determining a mix of use-cases of the candidate software system exercised by the intercepted request or an aggregate code coverage of the candidate software system of the intercepted request, and

wherein the intercepting is performed at least until one or more conditions regarding the mix of use-cases of the candidate software system exercised by the intercepted request or the aggregate code coverage of the intercepted request are satisfied.

10. The computer-implemented method as recited in claim 6, wherein the production software system modifies system data of a production system and the at least one candidate request is processed by the candidate software system such that system data of the production system is not modified.

11. The computer-implemented method as recited in claim 6, wherein the generating the validation result identifies differences specified as being unacceptable to the operation of the candidate software system.

12. The computer-implemented method as recited in claim 11, further comprising identifying one or more code changes between a current version of the candidate software system and a prior version of the candidate software system that resulted in the differences based at least in part on information from a source control system of the candidate software system.

13. The computer-implemented method as recited in claim 6, wherein the generating the validation result identifies unexpected differences and suppresses expected differences between the at least one respective candidate response and at least one respective production response.

14. The computer-implemented method as recited in claim 6, wherein a delay occurs between an intercepting of the at least one intercepted request and the requesting the candidate software system to process the at least one candidate request.

15. The computer-implemented method as recited in claim 6, wherein the candidate software system is a candidate version of the production software system.

16. The computer-implemented method as recited in claim 6, wherein at least one of:

the candidate software system comprises a different interface from that of the production software system;

the candidate software system comprises a different number of services or components from the production software system;

the candidate software system comprises a different framework or a different protocol from that of the production software system;

or

the candidate software system operates in a different operating system or environment from the production software system.

17. One or more non-transitory computer-readable storage media storing computer-executable instructions that are executable in at least one computing device to perform operations comprising:

selecting one or more intercepted requests logged by a duplicating proxy service to be replayed by the duplicating proxy service by reprocessing the one or more intercepted requests without modifying production system data of a production system, the one or more intercepted requests corresponding to one or more production requests that are intercepted based, at least in part, on one or more sampling rules prior to processing by the production system;

replaying at least one intercepted request, of the one or more intercepted requests, to a candidate software system and an authority software system based at least part on the selecting; and

generating performance metrics for the candidate software system, the candidate software system being a test software system that is a modified version of the production system that is to be validated or a replacement for the production system, the candidate software system operating without modifying the production system data, the performance metrics including at least one metric relating to differences between a candidate response generated by the candidate software system and an authority response generated by the authority software system that is a software system being used for validating the candidate software system, the candidate software system being a different software system from the production software system and the authority software system.

18. The one or more non-transitory computer-readable media as recited in claim 17, wherein at least one of:

the candidate software system comprises a different interface from that of the production software system or the authority software system;

the candidate software system comprises a different number of services or components from the production software system or the authority software system;

the candidate software system comprises a different framework or a different protocol from that of the production software system or the authority software system;

or

the candidate software system operates in a different operating system or environment from the production software system or the authority software system.

19. The one or more non-transitory computer-readable media as recited in claim 17, wherein the selecting is based at least in part on at least one of:

a specified type of difference between the candidate response and the authority response;

one or more specified use-cases of the candidate software system;

or

one or more specified code paths of the candidate software system.

20. The one or more non-transitory computer-readable media as recited in claim 17, the operations further comprising:

receiving an indication of one or more portions of the candidate response; and

configuring the duplicating proxy service to compare and analyze the one or more portions of the candidate response with at least one respective authority response.