Application experiment system
First Claim
1. A method for performing an experiment on a software application, the method comprising:
- under control of a computing system comprising computing hardware including one or more processors;
during execution of an instance of a software application on the computing system, receiving, from a remote system, an allocation of a treatment configuration for an experiment for the software application, wherein the treatment configuration includes;
at least one test factor;
at least one test value corresponding to the at least one test factor; and
a treatment distribution parameter indicating how the treatment configuration is to be distributed to one or more computing systems executing instances of the software application, wherein the treatment distribution parameter specifies a percent value indicating a percentage of running instances of the software application that should receive the treatment configuration, and wherein the treatment configuration is allocated based on the treatment distribution parameter;
receiving the at least one test value corresponding to the at least one test factor associated with the treatment configuration; and
modifying the instance of the software application to reference the at least one test value during execution.
0 Assignments
0 Petitions
Accused Products
Abstract
Examples of an application experiment system are provided that can manage and run experiments for software applications. In one embodiment, the application experiment system receives definitions for experiments and treatments for an application, runs experiments in response to requests from users of the system, automatically allocates treatments to users of the application, logs experiment data and treatment allocation data, and/or reports experiment results data. The application experiment system enables application developers to test changes or features in a precise, controlled environment on a limited user base before deciding whether to deploy changes or features to all users of an application.
-
Citations
17 Claims
-
1. A method for performing an experiment on a software application, the method comprising:
under control of a computing system comprising computing hardware including one or more processors; during execution of an instance of a software application on the computing system, receiving, from a remote system, an allocation of a treatment configuration for an experiment for the software application, wherein the treatment configuration includes; at least one test factor; at least one test value corresponding to the at least one test factor; and a treatment distribution parameter indicating how the treatment configuration is to be distributed to one or more computing systems executing instances of the software application, wherein the treatment distribution parameter specifies a percent value indicating a percentage of running instances of the software application that should receive the treatment configuration, and wherein the treatment configuration is allocated based on the treatment distribution parameter; receiving the at least one test value corresponding to the at least one test factor associated with the treatment configuration; and modifying the instance of the software application to reference the at least one test value during execution. - View Dependent Claims (2, 3, 4, 5)
-
6. A system comprising:
-
a non-transitory electronic data store configured to at least store an experiment configuration specifying one or more treatment configurations for a software testing experiment for a software application, wherein individual ones of the treatment configurations specify at least one test factor, at least one corresponding test value, and a treatment distribution parameter indicating how the individual treatment configuration is to be distributed to one or more remote computing devices executing instances of the software application, wherein the treatment distribution parameter specifies a percent value indicating a percentage of running instances of the software application that should receive the individual treatment configuration; and a computing system comprising one or more hardware computing devices, said computing system in communication with the electronic data store and configured to at least; receive, from a remote computing device executing an instance of the software application, a request to allocate a treatment configuration for the software testing experiment; retrieve the experiment configuration from the electronic data store; allocate, based at least in part on the experiment configuration and associated treatment distribution parameters, a selected treatment configuration to the remote computing device; and provide, to the remote computing device, the at least one test value associated with the selected treatment configuration, wherein the instance of the software application is modified to reference the at least one test value during execution of the software application. - View Dependent Claims (7, 8, 9, 10, 11)
-
-
12. Non-transitory computer-readable storage comprising computer-executable instructions stored thereon that, when executed by one or more hardware processors, are configured to cause the one or more hardware processors to perform operations comprising:
-
receiving, from a remote computing device executing an instance of a software application, a request to allocate a treatment configuration for a software testing experiment; accessing, by a computing system comprising one or more hardware computing devices, an experiment configuration from an electronic data store comprising hardware, wherein the experiment configuration specifies one or more treatment configurations, and wherein individual ones of the treatment configurations specify at least one test factor, at least one corresponding test value, and a treatment distribution parameter indicating how the individual treatment configuration is to be distributed to one or more remote computing devices executing instances of the software application, wherein the treatment distribution parameter specifies a percent value indicating a percentage of running instances of the software application that should receive the individual treatment configuration; allocating, based at least in part on the experiment configuration and associated treatment distribution parameters, a selected treatment configuration to the remote computing device; and providing, to the remote computing device, the at least one test value associated with the selected treatment configuration, wherein the instance of the software application is modified to reference the at least one test value during execution of the software application. - View Dependent Claims (13, 14, 15, 16, 17)
-
Specification