Shadow test replay service

• US 9,672,137 B1
• Filed: 03/06/2015
• Issued: 06/06/2017
• Est. Priority Date: 09/14/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Ramalingam, Harsha, Allocca, William Wallace, Telang, Nandan U., Bodley, James Cade, Cherukuri, Ram, Masilamani, Bharathi
• Primary Examiner(s): Dao, Thuy
• Assistant Examiner(s): BERMAN, STEPHEN DAVID
• Application Number:
  US14/641,177
• Time in Patent Office: 823 Days
• Field of Search: None
• CPC Class Codes: G06F 11/3457 Performance evaluation by simulation
• CPC Class Codes: G06F 11/3461 Trace driven simulation
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

The techniques described herein provide software testing of a candidate version of software. In some examples, an interceptor intercepts at least one production request to a production version of the software and issues the production request to a shadow proxy service as a shadow request. The shadow proxy service causes the at least one shadow request to be processed by the candidate version of the software being validated and an authority version of the software being used to validate the candidate version. The shadow proxy service may then compare and/or analyze at least one candidate response to the shadow request from the candidate version and at least one authority response to the shadow request from the authority version. A dashboard service may provide at least some of the resulting information and issue a request the shadow proxy service to replay at least one of the shadow requests.

Claims

1. One or more non-transitory computer-readable storage media storing computer-executable instructions that, when executed on one or more processors, perform acts comprising:

receiving one or more test scenarios, individual test scenarios of the one or more test scenarios corresponding to a use case for a candidate version of a software, wherein the one or more test scenarios are defined by data associated with production requests associated with a production system;

logging, in a storage device, one or more shadow requests processed by a shadow proxy service, the logging to populate indexed fields with data associated with the one or more shadow requests, the one or more shadow requests corresponding to one or more respective production requests that are intercepted prior to processing by the production system;

mapping individual shadow requests of the one or more shadow requests to one or more test scenarios of the one or more test scenarios based at least in part on the indexed fields associated with the individual shadow requests;

selecting a subset of the one or more shadow requests based at least in part on the data in the indexed fields to be replayed by the shadow proxy service; and

reprocessing at least one shadow request of the subset of the one or more shadow requests without modifying production system data.

2. The one or more non-transitory computer-readable media as recited in claim 1, the acts further comprising:

receiving system performance metrics for the candidate version of the software, the candidate version being a test version of a production version of the software of the production system, the candidate version operating without modifying system data, the system performance metrics including at least one metric relating to differences between a candidate response generated by the candidate version and an authority response generated by the production version of the software.

3. The one or more non-transitory computer-readable media as recited in claim 1, wherein the selecting the subset of the one or more shadow requests further comprises performing a search of a repository of logged shadow requests stored in a searchable catalog organized in a hierarchical manner.

4. The one or more non-transitory computer-readable media as recited in claim 3, wherein the search returns logged requests that are associated with at least one of:

a specified type of difference between a candidate response generated by a candidate version of a software and an authority response generated by a production version of the software;

one or more specified use-cases associated with the software;

or

one or more specified code paths associated with the candidate version.

5. The non-transitory one or more computer-readable media as recited in claim 1, the acts further comprising:

receiving a selection of a shadow response to one of the one or more shadow requests; and

configuring the shadow proxy service to compare the shadow response with a respective authority response that modified the production system data.

6. The one or more non-transitory computer-readable media as recited in claim 1, wherein the shadow proxy service is configured for execution during execution of the production system.

7. The one or more non-transitory computer-readable media as recited in claim 1, the acts further comprising:

causing the shadow service to terminate replay of the at least one of the subset of the one or more shadow requests in response to a threshold amount of shadow responses being different than respective production responses.

19. The one or more non-transitory computer-readable media as recited in claim 1, wherein selecting the subset of the one or more shadow requests is further based at least in part on determining an amount of shadow requests to be reprocessed to achieve a confidence level threshold.

20. The one or more non-transitory computer-readable media as recited in claim 1, wherein the shadow proxy service processing an individual shadow request of the one or more shadow request comprises:

causing, by the shadow proxy service, the individual shadow request to be processed by the candidate version of the software, and

causing, by the shadow proxy service, the individual shadow request to be processed by an authority version of the software.

8. A system comprising:

one or more processors;

memory storing computer-executable instructions that, when executed, cause the one or more processors to;

store one or more test scenarios, individual test scenarios of the one or more test scenarios corresponding to a use case for a candidate version of a software, wherein the one or more test scenarios are defined based at least in part by data associated with production requests associated with a production system;

log, in a repository, one or more shadow requests processed by a shadow proxy service, the logging comprising populating indexed fields with data associated with the one or more shadow requests, the one or more shadow requests corresponding to one or more respective production requests that are intercepted prior to processing by the production system;

map individual ones of the one or more shadow requests to the one or more test scenarios using the indexed fields;

select, based at least in part on the data in the indexed fields, a subset of the one or more shadow requests to be replayed by the shadow proxy service by reprocessing at least one of the one or more production requests without modifying production system data; and

replay at least one of the shadow requests of the subset of the one or more shadow requests based at least part on the selecting.

9. The system as recited in claim 8, wherein the computer-executable instructions further cause the one or more processors to receive system performance metrics for a candidate version of a software, the candidate version being a test version of a production version of the software of the production system, the candidate version operating without modifying system data.

10. The system as recited in claim 9, wherein the system performance metrics comprise at least one metric associated with differences between a candidate response generated by the candidate version and an authority response generated by the production version of the software.

11. The system as recited in claim 8, wherein the repository of the logged one or more shadow requests comprises a searchable catalog organized in a hierarchical manner.

12. The system as recited in claim 8, wherein the computer-executable instructions further cause the one or more processors to perform the replay at least partially overlapping in time with execution of the production system.

13. The system as recited in claim 8, wherein the computer-executable instructions further cause the one or more processors to terminate replay of the at least one of the shadow requests of the subset of the one or more shadow requests in response to a threshold amount of shadow responses being different than respective production responses.

14. The system as recited in claim 8, wherein the computer-executable instructions further cause the one or more processors to:

receive a selection of a shadow response to one of the one or more shadow requests; and

configuring the shadow proxy service to compare the shadow response with a respective authority response that modified the production system data.

15. A computer-executable method comprising:

receiving one or more test scenarios, individual test scenarios of the one or more test scenarios corresponding to a use case for a candidate version of a software, wherein the one or more test scenarios are defined by data associated with production requests associated with a production system;

logging, in a storage device, one or more shadow requests processed by a shadow proxy service executed by one or more computing devices, the logging to populate indexed fields with data associated with the one or more shadow requests, the one or more shadow requests corresponding to one or more respective production requests that are intercepted prior to processing by the production system;

mapping individual shadow requests of the one or more shadow requests to the one or more test scenarios based at least in part on the indexed fields associated with the individual shadow requests;

selecting, by the shadow proxy service, a subset of the one or more shadow requests to be replayed by the shadow proxy service based at least in part on the data in the indexed fields; and

reprocessing at least one of the subset of the one or more shadow requests without modifying production system data.

16. The computer-executable method as recited in claim 15, further comprising logging a plurality of production requests processed by the production system in a repository for use in the selecting.

17. The computer-executable method as recited in claim 15, further comprising receiving system performance metrics for a candidate version of a software, the candidate version being a test version of a production version of the software of the production system, the candidate version operating without modifying system data, the system performance metrics including differences between a candidate response generated by the candidate version and an authority response generated by the production version of the software.

18. The computer-executable method as recited in claim 15, further comprising causing the shadow service to terminate replay of the at least one of the subset of the one or more shadow requests in response to a threshold amount of shadow responses being different than respective production responses.