Application experiment system

• US 9,665,475 B2
• Filed: 07/02/2014
• Issued: 05/30/2017
• Est. Priority Date: 06/07/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Suttle, Ian C., Lee, Steven J., King, Marcus A., Berger, Timothy J.
• Primary Examiner(s): Chavis, John
• Application Number:
  US14/322,731
• Publication Number:
  US 20150007135A1
• Time in Patent Office: 1,063 Days
• Field of Search: 717121
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/34 involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416

Abstract

Examples of an application experiment system are provided that can manage and run experiments for software applications. In one embodiment, the application experiment system receives definitions for experiments and treatments for an application, runs experiments in response to requests from users of the system, automatically allocates treatments to users of the application, logs experiment data and treatment allocation data, and/or reports experiment results data. The application experiment system enables application developers to test changes or features in a precise, controlled environment on a limited user base before deciding whether to deploy changes or features to all users of an application.

Claims

1. A method for performing an experiment on a software application, the method comprising:

under control of a computing system comprising computing hardware including one or more processors;

during execution of an instance of a software application on the computing system, receiving, from a remote system, an allocation of a treatment configuration for an experiment for the software application, wherein the treatment configuration includes;

at least one test factor;

at least one test value corresponding to the at least one test factor; and

a treatment distribution parameter indicating how the treatment configuration is to be distributed to one or more computing systems executing instances of the software application, wherein the treatment distribution parameter specifies a percent value indicating a percentage of running instances of the software application that should receive the treatment configuration, and wherein the treatment configuration is allocated based on the treatment distribution parameter;

receiving the at least one test value corresponding to the at least one test factor associated with the treatment configuration; and

modifying the instance of the software application to reference the at least one test value during execution.

2. The method of claim 1, wherein the treatment configuration allocated to the computing system is different from a previous treatment configuration that was allocated to the computing system during a previous allocation.

3. The method of claim 1, wherein the at least one test value associated with the at least one test factor is updated during execution of the instance of the software application.

4. The method of claim 1, further comprising:

receiving, from the remote system, an allocation of a different treatment configuration for the experiment for the software application; and

receiving at least one test value corresponding to at least one test factor associated with the different treatment configuration.

5. The method of claim 1, further comprising:

logging treatment data associated with the at least one test value and the modified instance of the software application; and

transmitting the treatment data to the remote system.

6. A system comprising:

a non-transitory electronic data store configured to at least store an experiment configuration specifying one or more treatment configurations for a software testing experiment for a software application,

wherein individual ones of the treatment configurations specify at least one test factor, at least one corresponding test value, and a treatment distribution parameter indicating how the individual treatment configuration is to be distributed to one or more remote computing devices executing instances of the software application, wherein the treatment distribution parameter specifies a percent value indicating a percentage of running instances of the software application that should receive the individual treatment configuration; and

a computing system comprising one or more hardware computing devices, said computing system in communication with the electronic data store and configured to at least;

receive, from a remote computing device executing an instance of the software application, a request to allocate a treatment configuration for the software testing experiment;

retrieve the experiment configuration from the electronic data store;

allocate, based at least in part on the experiment configuration and associated treatment distribution parameters, a selected treatment configuration to the remote computing device; and

provide, to the remote computing device, the at least one test value associated with the selected treatment configuration, wherein the instance of the software application is modified to reference the at least one test value during execution of the software application.

7. The system of claim 6, wherein the selected treatment configuration allocated to the remote computing device is different from a previous treatment configuration that was allocated to the remote computing device in response to a previous request to allocate.

8. The system of claim 6, wherein the computing system is further configured to receive, from the remote computing device, treatment data associated with the selected treatment configuration and the modified instance of the software application.

9. The system of claim 6, wherein to allocate a selected treatment configuration to the remote computing device, the computing system is further configured to:

determine a unique identifier associated with the remote computing device;

generate, via a hashing algorithm, a hashed value based on the unique identifier, wherein the hashed value is a numeric value in the range of 0 to 99; and

allocate the selected treatment configuration based on a comparison of the associated treatment distribution parameters for the experiment configuration to the hashed value.

10. The system of claim 6, wherein the computing system is further configured to:

receive an updated treatment distribution parameter for the selected treatment configuration;

allocate, to the remote computing device, a different treatment configuration associated with the experiment configuration, based at least in part on the updated treatment distribution parameter; and

provide, to the remote computing device, the at least one test value associated with the different treatment configuration.

11. The system of claim 10, wherein the updated treatment distribution parameter is received via a user interface configured to receive the experiment configuration for the software testing experiment.

12. Non-transitory computer-readable storage comprising computer-executable instructions stored thereon that, when executed by one or more hardware processors, are configured to cause the one or more hardware processors to perform operations comprising:

receiving, from a remote computing device executing an instance of a software application, a request to allocate a treatment configuration for a software testing experiment;

accessing, by a computing system comprising one or more hardware computing devices, an experiment configuration from an electronic data store comprising hardware, wherein the experiment configuration specifies one or more treatment configurations, and wherein individual ones of the treatment configurations specify at least one test factor, at least one corresponding test value, and a treatment distribution parameter indicating how the individual treatment configuration is to be distributed to one or more remote computing devices executing instances of the software application, wherein the treatment distribution parameter specifies a percent value indicating a percentage of running instances of the software application that should receive the individual treatment configuration;

allocating, based at least in part on the experiment configuration and associated treatment distribution parameters, a selected treatment configuration to the remote computing device; and

providing, to the remote computing device, the at least one test value associated with the selected treatment configuration,

wherein the instance of the software application is modified to reference the at least one test value during execution of the software application.

13. The non-transitory computer-readable storage of claim 12, wherein allocating, based at least in part on the experiment configuration and associated treatment distribution parameters, a selected treatment configuration to the remote computing device further comprises:

determining a unique identifier associated with the remote computing device;

generating, via a hashing algorithm, a hashed value based on the unique identifier, wherein the hashed value is a numeric value in the range of 0 to 99; and

allocating the selected treatment configuration based on a comparison of the associated treatment distribution parameters for the experiment configuration to the hashed value.

14. The non-transitory computer-readable storage of claim 12, wherein the operations further comprise:

receiving an updated treatment distribution parameter for the selected treatment configuration;

allocating, to the remote computing device, a different treatment configuration associated with the experiment configuration, based at least in part on the updated treatment distribution parameter; and

providing, to the remote computing device, the at least one test value associated with the different treatment configuration.

15. The non-transitory computer-readable storage of claim 14, wherein the updated treatment distribution parameter is received via a user interface configured to receive the experiment configuration for the software testing experiment.

16. The non-transitory computer-readable storage of claim 12, wherein the operations further comprise receiving, from the remote computing device, treatment data associated with the selected treatment configuration and the modified instance of the software application.

17. The non-transitory computer-readable storage of claim 12, wherein the selected treatment configuration allocated to the remote computing device is different from a previous treatment configuration that was allocated to the remote computing device in response to a previous request to allocate.