Monitoring landing page experiments

• US 20070271352A1
• Filed: 05/22/2006
• Published: 11/22/2007
• Est. Priority Date: 05/22/2006
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Vasilik, Kenneth Eric, Lin, Zhen, Bardin, Ariel, Khopkar, Chirag, Nelson, Dorothy Ann
• Granted Patent: US 8,682,712 B2
• Time in Patent Office: Days
• US Class Current: 709/217
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

Methods, systems, and apparatus, including computer program products for monitoring an experiment relating to the conversion rates of landing page variants to test landing page designs. One method includes computing respective conversion rates for each of a plurality of corresponding landing page variants, based on the number of times a conversion page is reached from the corresponding landing page variant, and based on the number of times the corresponding landing page variant is reached from one or more advertisements; and computing a confidence interval for each conversion rate. A user monitoring the experiment can be notified if winning or losing variants emerge. A time to completion of the experiment can be estimated. Presentation to browsing users of winning variants can be favored, and losing variants can be disfavored, during the experiment as they emerge.

Claims

1. A computer-implemented method comprising:

computing respective conversion rates for each of a plurality of corresponding landing page variants, based on the number of times a conversion page is reached from the corresponding landing page variant, and based on the number of times the corresponding landing page variant is reached from one or more advertisements, wherein;

a landing page comprises one or more sections, there being for each of the sections a corresponding plurality of alternative versions;

a combination of alternative versions corresponds to each landing page variant, the combination associating one alternative version with each of the sections; and

each landing page variant comprises the landing page with each of the one or more sections having the version associated with the section by the combination; and

computing a confidence interval for each conversion rate.

2. The method of claim 1, wherein:

the method is performed by software running on one or more servers.

3. The method of claim 1, further comprising:

identifying two landing page variants with the highest conversion rates as winning candidates;

determining whether the confidence intervals for the winning candidates are non-overlapping; and

notifying a monitoring user of the combination of alternative versions with the highest conversion rate if the confidence intervals for the winning candidates are non-overlapping.

4. The method of claim 3, further comprising:

determining whether the confidence intervals for the winning candidates are non-overlapping and if so automatically ending the experiment.

5. The method of claim 1, further comprising:

identifying as losing candidates any landing page variants having confidence intervals that are non-overlapping from and less than the confidence interval for a baseline landing page variant; and

notifying a monitoring user of any losing candidates.

6. The method of claim 5, further comprising:

automatically black-listing any losing candidates so that they are no longer tested.

7. The method of claim 1, further comprising:

identifying one or more landing page variants with the highest conversion rates;

identifying one or more landing page variants with the lowest conversion rates; and

automatically configuring a landing page server serving the landing page so that the choosing of alternative versions favors the landing page variants with the highest conversion rates and disfavors the landing page variants with the lowest conversion rates.

8. The method of claim 1, further comprising:

estimating a time to completion based on the specified number of alternative versions for each section, an assumed frequency of how often the landing page will be reached, a number of times the landing page has been reached, a baseline percentage of conversions, and an assumed improvement in the percentage of conversions, wherein completion is deemed to have occurred when the confidence intervals for the two landing page variants with the highest conversion rates become non-overlapping.

9. The method of claim 1, further comprising:

estimating a time to completion based on a projected frequency of how often two landing page variants with the highest conversion rates will be reached, a number of times the two landing page variants have been reached, and the percentage of conversions for both of the two landing page variants, wherein completion is deemed to have occurred when the confidence intervals for the two landing page variants become non-overlapping.

10. A system comprising:

means for computing respective conversion rates for each of a plurality of corresponding landing page variants, based on the number of times a conversion page is reached from the corresponding landing page variant, and based on the number of times the corresponding landing page variant is reached from one or more advertisements, wherein;

a landing page comprises one or more sections, there being for each of the sections a corresponding plurality of alternative versions;

a combination of alternative versions corresponds to each landing page variant, the combination associating one alternative version with each of the sections; and

each landing page variant comprises the landing page with each of the one or more sections having the version associated with the section by the combination; and

means for computing a confidence interval for each conversion rate.

11. The system of claim 10, wherein:

the means for computing respective conversion rates comprises software running on one or more servers; and

the means for computing a confidence interval for each conversion rate comprises software running on one or more servers.

12. The system of claim 10, further comprising:

means for identifying two landing page variants with the highest conversion rates as winning candidates;

means for determining whether the confidence intervals for the winning candidates are non-overlapping; and

means for notifying a monitoring user of the combination of alternative versions with the highest conversion rate if the confidence intervals for the winning candidates are non-overlapping.

13. The method of claim 12, further comprising:

means for determining whether the confidence intervals for the winning candidates are non-overlapping and if so automatically ending the experiment.

14. The system of claim 10, further comprising:

means for identifying as losing candidates any landing page variants having confidence intervals that are non-overlapping from and less than the confidence interval for a baseline landing page variant; and

means for notifying a monitoring user of any losing candidates.

15. The method of claim 14, further comprising:

means for automatically black-listing any losing candidates so that they are no longer tested.

16. The system of claim 10, further comprising:

means for identifying one or more landing page variants with the highest conversion rates;

means for identifying one or more landing page variants with the lowest conversion rates; and

means for automatically configuring a landing page server serving the landing page so that the choosing of alternative versions favors the landing page variants with the highest conversion rates and disfavors the landing page variants with the lowest conversion rates.

17. The system of claim 10, further comprising:

means for estimating a time to completion based on the specified number of alternative versions for each section, an assumed frequency of how often the landing page will be reached, a number of times the landing page has been reached, a baseline percentage of conversions, and an assumed improvement in the percentage of conversions, wherein completion is deemed to have occurred when the confidence intervals for the two landing page variants with the highest conversion rates become non-overlapping.

18. The system of claim 10, further comprising:

means for estimating a time to completion based on a projected frequency of how often two landing page variants with the highest conversion rates will be reached, a number of times the two landing page variants have been reached, and the percentage of conversions for both of the two landing page variants, wherein completion is deemed to have occurred when the confidence intervals for the two landing page variants become non-overlapping.

19. A computer program product, encoded on a computer-readable medium, operable to cause data processing apparatus to perform operations comprising:

computing respective conversion rates for each of a plurality of corresponding landing page variants, based on the number of times a conversion page is reached from the corresponding landing page variant, and based on the number of times the corresponding landing page variant is reached from one or more advertisements, wherein;

a landing page comprises one or more sections, there being for each of the sections a corresponding plurality of alternative versions;

a combination of alternative versions corresponds to each landing page variant, the combination associating one alternative version with each of the sections; and

each landing page variant comprises the landing page with each of the one or more sections having the version associated with the section by the combination; and

computing a confidence interval for each conversion rate.

20. The product of claim 19, further operable to cause data processing apparatus to perform operations comprising:

identifying two landing page variants with the highest conversion rates as winning candidates;

determining whether the confidence intervals for the winning candidates are non-overlapping; and

notifying a monitoring user of the combination of alternative versions with the highest conversion rate if the confidence intervals for the winning candidates are non-overlapping.

21. The product of claim 20, further operable to cause data processing apparatus to perform operations comprising:

determining whether the confidence intervals for the winning candidates are non-overlapping and if so automatically ending the experiment.

22. The product of claim 19, further operable to cause data processing apparatus to perform operations comprising:

identifying as losing candidates any landing page variants having confidence intervals that are non-overlapping from and less than the confidence interval for a baseline landing page variant; and

notifying a monitoring user of any losing candidates.

23. The product of claim 22, further operable to cause data processing apparatus to perform operations comprising:

automatically black-listing any losing candidates so that they are no longer tested.

24. The product of claim 19, further operable to cause data processing apparatus to perform operations comprising:

identifying one or more landing page variants with the highest conversion rates;

identifying one or more landing page variants with the lowest conversion rates; and

automatically configuring a landing page server serving the landing page so that the choosing of alternative versions favors the landing page variants with the highest conversion rates and disfavors the landing page variants with the lowest conversion rates.

25. The product of claim 19, further operable to cause data processing apparatus to perform operations comprising:

estimating a time to completion based on the specified number of alternative versions for each section, an assumed frequency of how often the landing page will be reached, a number of times the landing page has been reached, a baseline percentage of conversions, and an assumed improvement in the percentage of conversions, wherein completion is deemed to have occurred when the confidence intervals for the two landing page variants with the highest conversion rates become non-overlapping.

26. The product of claim 19, further operable to cause data processing apparatus to perform operations comprising:

estimating a time to completion based on a projected frequency of how often two landing page variants with the highest conversion rates will be reached, a number of times the two landing page variants have been reached, and the percentage of conversions for both of the two landing page variants, wherein completion is deemed to have occurred when the confidence intervals for the two landing page variants become non-overlapping.