Adaptive website optimization experiment

• US 8,234,632 B1
• Filed: 10/22/2007
• Issued: 07/31/2012
• Est. Priority Date: 10/22/2007
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Vasilik, Kenneth Eric, Meyer, Michael M., Hugeback, Angela B.
• Primary Examiner(s): Zhen, Wei
• Assistant Examiner(s): Wu, Junchun
• Application Number:
  US11/876,221
• Time in Patent Office: 1,744 Days
• Field of Search: 717/130
• US Class Current: 717/130
• CPC Class Codes: G06F 11/3428 Benchmarking
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 2201/875 Monitoring of systems including the internet
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

Systems and methods for adaptive website optimization experiments. Adaptive website optimization can include collecting statistics associated with a number of landing page configurations and identifying a preferred group of landing page configurations during the pendency of an experiment based upon the collected statistics and preferentially selecting a landing page configuration from the preferred group of options for display by an instrumented web page associated with the experiment.

Claims

1. A system comprising:

one or more processors;

a computer-readable storage medium coupled to the one or more processors and having instructions stored thereon, which, when executed by the one or more processors, cause the one or more processors to perform operations to provide;

a website experiment interface operable to receive, from a publisher of a website, parameters for an experiment including a plurality of options, each option specifying a respective page configuration, and operable to provide a control script to the publisher based upon the options, the control script being operable to be inserted into a landing page to provide an instrumented landing page;

a statistics module operable to collect performance statistics associated with each option, the performance statistics measuring a performance of the instrumented landing page in each respective page configuration served to a client device, the statistics module being further operable to analyze the performance statistics to identify a preferred group of options and a non-preferred group of options, the options of the preferred group outperforming the options of the non-preferred group, the statistics module operable to;

produce adjusted statistics based upon;

adjusting the performance statistics based upon at least one of a first correction for performing multiple comparisons if there are more than two alternative landing page configurations, and a second correction for performing an interim analysis if the determination is being made prior to the scheduled end of the website optimization experiment, and

analyzing the adjusted statistics to identify one or more of the plurality of landing page configurations that is performing statistically significantly better than any other one or more of the plurality of landing page configurations; and

an option selection module operable to receive a communication from the client device based upon the client device loading the instrumented web page, the option selection module being operable to, in response to the communication, provide a page configuration corresponding to a selected option from the preferred group of options, the page configuration being provided to the instrumented landing page, the selected option being selected more frequently from the preferred group of options than from the non-preferred group of options.

2. The system of claim 1, wherein the statistics module is operable to defer identification of the preferred group of options and the non-preferred group of options until a threshold sample size is obtained.

3. The system of claim 1, wherein the statistics module is operable to update the preferred group of options and the non-preferred group of options throughout a duration of the experiment specified by the parameters associated with the experiment.

4. The system of claim 1, wherein the system is operable to reduce a cost associated with running the experiment, the cost comprising an opportunity cost associated with each time a low-performing option from among the plurality of options is served to the client device relative to a higher performing option from among the plurality of options.

5. The system of claim 1, wherein the website experiment interface is operable to receive a request to review current results during an experiment period associated with the experiment, the website experiment interface being operable to retrieve a current set of results from the statistics module, the statistics module being operable to adjust the current set of results using an alpha spending function.

6. A method comprising:

initiating a website optimization experiment, the website optimization experiment comprising a comparison of a plurality of alternative landing page configurations using an instrumented landing page;

collecting statistics associated with each of the alternative landing page configurations;

producing adjusted statistics by adjusting the statistics based upon at least one of a first correction for performing multiple comparisons if there are more than two alternative landing page configurations, and a second correction for performing an interim analysis if the determination is being made prior to the scheduled end of the website optimization experiment;

receiving a communication from a browser based upon the browser loading the instrumented landing page;

analyzing, in response to the communication, the adjusted statistics to identify one or more of the plurality of alternative landing page configurations that is performing statistically significantly better than any other one or more of the plurality of alternative landing page configurations; and

preferentially providing a selected landing page configuration to the instrumented landing page based upon an analysis of the adjusted statistics, the selected landing page configuration being selected more frequently from a preferred group of alternative landing page configurations than from the non-preferred group of alternative landing page configurations.

7. The method of claim 6, wherein producing adjusted statistics comprises adjusting the statistics using a multiple comparison correction algorithm.

8. The method of claim 7, wherein producing adjusted statistics comprises adjusting the statistics using an alpha spending function prior to determining whether any of the plurality of alternative landing page configurations is currently forecasted to provide better results than the non-preferred group of alternative landing page configurations.

9. The method of claim 6, further comprising deferring a determination of the preferred group of alternative landing page configurations and the non-preferred group of alternative landing page configurations until a threshold sample size is obtained.

10. The method of claim 6, further comprising updating the preferred group of alternative landing page configurations and the non-preferred group of alternative landing page configurations throughout a duration associated with the website optimization experiment.

11. The method of claim 6, further comprising reducing a cost associated with running the website optimization experiment, the cost comprising an opportunity cost associated with each time a low-performing alternative landing page configuration from among the plurality of alternative landing page configurations is served to a client device relative to a higher performing alternative landing page configuration from among the plurality of alternative landing page configuration.

12. The method of claim 6, further comprising:

receiving a request to review current results during an experiment period associated with the website optimization experiment;

retrieving the current results based upon the statistics; and

providing the current results to a publisher associated with the website optimization experiment.

13. One or more computer readable media operable to select a landing page configuration from among a plurality of alternative landing page configurations and coupled to one or more processors having instructions stored thereon, that when executed by the one or more processors, cause the one or more processors to perform operations, comprising:

initiating a website optimization experiment to determine whether a particular landing page configuration from among the plurality of alternative landing page configurations including a current landing page configuration performs better than any other alternative landing page configurations;

collecting statistics related to a performance of each of the plurality of alternative landing pages configurations;

producing adjusted statistics by adjusting the statistics based upon at least one of a first correction for performing multiple comparisons if there are more than two alternative landing page configurations, and a second correction for performing an interim analysis if the determination is being made prior to the scheduled end of the website optimization experiment;

receiving a communication from a client device responsive to the client device loading an instrumented landing page comprising the plurality of alternative landing page configurations, the communication comprising a request to identify a selected landing page configuration from the plurality of alternative landing page configurations;

determining a preferred group of landing page configurations based upon analyzing the adjusted statistics, the preferred group of landing page configurations comprising one or more landing page configurations that are performing statistically significantly better than a non-preferred group of landing page configurations; and

providing, in response to the communication, a selected landing page configuration to the instrumented landing page based upon a determination of the preferred group of landing page configurations, the selected landing page configuration being selected from the preferred group of landing page configurations more frequently than from the non-preferred group of landing page configurations.

14. The computer readable media of claim 13, further comprising deferring determining a preferred group of landing page configurations until a reasonable sample size has been collected.

15. A method comprising:

collecting statistics associated with each of a plurality of alternative landing page configurations;

producing adjusted statistics by adjusting the statistics based upon at least one of a first correction for performing multiple comparisons if there are more than two alternative landing page configurations, and a second correction for performing an interim analysis if the determination is being made prior to a scheduled end of the website optimization experiment;

analyzing the adjusted statistics to identify a preferred group of landing page configurations and a non-preferred group of landing page configurations from the plurality of alternative landing page configurations, the landing page configurations of the preferred group providing statistically significantly better results than the landing page configurations of the non-preferred group; and

preferentially providing a landing page configuration to an instrumented landing page based upon the statistics, the landing page configuration being selected more frequently from the preferred group of landing page configurations than from the non-preferred group of landing page configurations.

16. The method of claim 15, further comprising reducing a cost associated with executing the method, the cost comprising an opportunity cost associated with each time a low-performing alternative landing page configuration from among the plurality of alternative landing page configurations is served to a client device relative to a higher performing alternative landing page configuration from among the plurality of alternative landing page configurations.

17. A method comprising:

initiating a website experiment comprising a comparison of a plurality of alternative landing page configurations, the website experiment having an experiment duration provided by a publisher of a website and being operable to determine a performance associated with each of the plurality of alternative landing page configurations;

collecting statistics associated with each of the plurality of alternative landing page configurations;

producing adjusted statistics by adjusting the statistics based upon at least one of a first correction for performing multiple comparisons if there are more than two alternative landing page configurations, and a second correction for performing an interim analysis if the determination is being made prior to a scheduled end of the website optimization experiment;

analyzing the adjusted statistics to identify one or more of the plurality of landing page configurations that is performing statistically significantly better than any other one or more of the plurality of landing page configurations; and

determining whether to end the website experiment based upon the adjusted statistics.