Running multiple web page experiments on a test page

• US 8,296,643 B1
• Filed: 10/18/2007
• Issued: 10/23/2012
• Est. Priority Date: 10/18/2007
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Vasilik, Kenneth Eric
• Primary Examiner(s): Huynh, Cong-Lac
• Application Number:
  US11/874,852
• Time in Patent Office: 1,832 Days
• Field of Search: 715/200, 715/255, 715/201, 715/202, 715/205
• US Class Current: 715/200
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking

Abstract

Methods, systems and apparatus, including computer program products, for performing multiple tests on a test web page. Source of a test web page is configured to perform multivariate testing by placing a number of variable content sections into the source of the test web page. Multiple tests having differing variable content for insertion into the variable content sections, or a subset thereof, can be performed without the need for the test page to be re-modified.

Claims

1. A method for delivering content combinations to a test web page instrumented for use with multiple experiments of a multivariate experimentation system, comprising:

receiving a page identifier of the test web page, the test web page having code defining a plurality of variable content sections, each variable content section of the plurality of variable content sections having a corresponding variable content section identifier;

receiving a first experiment schema, the first experiment schema including a first set of identifiers associated with a first subset of variable content sections of the plurality of variable content sections;

receiving a second experiment schema, the second experiment schema including a second set of identifiers associated with a second subset of variable content sections of the plurality of variable content sections;

retrieving, from a plurality of experiment identifiers associated with the page identifier, a first experiment identifier of a first experiment that corresponds to the first experiment schema;

retrieving a first set of variable content items based on the first experiment identifier and the first set of identifiers;

sending the first set of variable content items to a first web browser for insertion into the first subset of variable content section of the test web page corresponding to the first set of identifiers, wherein the first set of variable content items includes a first particular subset of variable content items that are associated with a first effectiveness in achieving a goal, the goal being selected based on different types of user interactions with the test web page;

retrieving, from the plurality of experiment identifiers associated with the page identifier, a second experiment identifier of a second experiment that corresponds to the second experiment schema;

retrieving a second set of variable content items based on the second experiment identifier and the second set of identifiers, wherein the second set of variable content items includes a second particular subset of variable content items that are associated with a second effectiveness in achieving the goal; and

sending the second variable content of the second experiment to a second web browser for insertion into the second subset of variable content sections of the test web page corresponding to the second set of identifiers.

2. The method of claim 1, wherein:

sending the first set of variable content items of the first experiment to a first web browser and sending the second set of variable content items of the second experiment to a second web browser are performed during a time in which the first set of variable content sections and the second set of variable content sections of the test web page code remain static.

3. The method of claim 1, wherein:

the first set of variable content items are identified from among a plurality of variable content items based on the first experiment identifier; and

the second set of variable content items are identified from among the plurality of variable content items based on the second experiment identifier.

4. The method of claim 1, further comprising:

receiving variable content section identifiers corresponding to the plurality of variable content sections of the test web page, wherein;

the first set of variable content items are identified from among the plurality of variable content items based on the first experiment identifier and the first set of identifiers; and

the second set of variable content items are identified from among the plurality of variable content items based on the second experiment identifier and the second set of identifiers.

5. The method of claim 1, wherein:

the first subset of variable content sections corresponds to the first set of identifiers stored in the multivariate experimentation system in association with the first experiment identifier; and

the second subset of variable content sections corresponds to the second set of identifiers stored in the multivariate experimentation system in association with the second experiment identifier.

6. The method of claim 1, wherein:

the first subset of variable content sections and the second subset of variable content sections have at least one variable content section in common.

7. The method of claim 1, wherein:

the first experiment includes variable content stored in the multivariate experimentation system to be presented in the first subset of variable content sections on the test web page over a first experiment period to test visitor response to viewing the first subset of variable content sections; and

the second experiment includes variable content stored in the multivariate experimentation system to be presented in the second subset of variable content sections on the test web page over a second experiment period to test visitor response to viewing the second subset of variable content sections.

8. The method of claim 7, wherein:

the first experiment period overlaps with the second experiment period.

9. The method of claim 1, further comprising:

measuring the first effectiveness of the test web page in achieving the goal, wherein the first effectiveness is based on the test web page displaying the first particular subset of variable content items associated with the first experiment identifier of the first experiment and the first set of identifiers;

measuring the second effectiveness of the test web page in achieving the goal, wherein the second effectiveness is based on the test web page displaying the second particular subset of variable content items associated with the second experiment identifier of the second experiment and the second set of identifiers; and

identifying the most effective website content based on the first effectiveness and the second effectiveness.

10. A method for delivering content combinations to a test web page instrumented for use with multiple experiments of a multivariate experimentation system, comprising:

receiving a page identifier of the test web page, the test web page having code defining a plurality of variable content sections, each variable content section of the plurality of variable content sections having a variable content section identifier, the variable content section identifier being provided by a name, number, or other descriptor;

receiving a first experiment schema, the first experiment schema including a first set of identifiers associated with a first subset of variable content sections of the plurality of variable content sections of the test web page and being one experiment schema of a plurality of experiment schemas;

identifying a first experiment of one or more experiments associated with the page identifier, the first experiment corresponding to the first experiment schema;

for respective variable content section identifiers in the first set of identifiers;

searching for variable content associated with the first experiment and the respective variable content section identifiers;

if variable content associated with the first experiment and the respective variable content section identifiers is found;

selecting respective variable content items associated with the first experiment and the respective variable content section identifiers; and

transmitting the respective variable content items for placement on the test web page in variable content sections corresponding to the respective variable content section identifiers, wherein the variable content items include a particular set of variable content items being associated with an effectiveness in achieving a goal, the goal being selected based on different types of user interactions with the test web page.

11. The method of claim 10, further comprising:

if variable content associated with the first experiment and the respective variable content section identifiers that have been received is not found, transmitting an instruction to display default content in the variable content sections corresponding to the respective variable content section identifiers that have been received.

12. The method of claim 10, further comprising:

if variable content associated with the first experiment and the respective variable content section identifiers that have been received is not found, proceeding to a next variable content section identifier that has been received.

13. The method of claim 10, further comprising:

identifying a second experiment of the one or more experiments associated with the page identifier, the second experiment corresponding to the second experiment schema;

for respective variable content section identifiers that have been received;

searching for variable content associated with the second experiment and the respective variable content section identifiers;

if variable content associated with the second experiment and the respective variable content section identifiers is found;

selecting respective variable content items associated with the second experiment and the respective variable content section identifiers; and

transmitting the respective variable content items for placement on the test web page in variable content sections corresponding to the respective variable content section identifiers.

14. The method of claim 13, further comprising:

measuring the effectiveness of the test web page in achieving the goal, wherein the effectiveness is based on the test web page displaying the particular set of variable content items associated with each of the first experiment and the second experiment and the respective variable content section identifiers that have been received; and

identifying the most effective website content based on the effectiveness determined from the first experiment and the effectiveness determined from the second experiment.

15. A multivariate experimentation system configured to perform multiple experiments on a test web page, comprising:

a user interface device; and

one or more computers operable to interact with the user interface device and to;

receive the test web page, the test web page having a page identifier;

identify a plurality of variable content sections included in the test web page, each variable content section having a corresponding variable content section identifier, a particular combination of variable content section identifiers being identified in an experiment schema associated with a particular experiment of the multiple experiments, a particular set of variable content items associated with the particular combination of variable content section identifiers being associated with an effectiveness in achieving a goal, the goal being selected based on different types of user interactions with the test web page;

send each corresponding variable content section identifier to the user interface device;

receive a first set of variable content section identifiers selected on the user interface device to define a first experiment schema associated with a first experiment to be conducted on the test web page, the first experiment including a first experiment identifier that is one identifier of a plurality of experiment identifiers associated with the test web page; and

receive a second set of variable content section identifiers selected on the user interface device to define a second experiment schema associated with a second experiment to be conducted on the test web page, the second experiment including a second experiment identifier that is one identifier of the plurality of experiment identifiers associated with the test web page.

16. The system of claim 15, wherein:

the first set of variable content section identifiers has at least one variable content section identifier in common with the second set of variable content section identifiers.

17. The system of claim 15, wherein the one or more computers are further operable to:

store the first experiment identifier of the first experiment and the second experiment identifier of the second experiment in association with the page identifier of the test web page.

18. The system of claim 15, wherein the one or more computers are further operable to:

receive the page identifier of the test web page;

receive each variable content section identifier;

identify the first experiment of one or more experiments associated with the page identifier, the first experiment being associated with the first set of variable content section identifiers;

for respective variable content section identifiers that have been received;

search for variable content associated with the first experiment and the respective variable content section identifiers;

if variable content associated with the first experiment and the respective variable content section identifiers is found;

select respective variable content items associated with the first experiment and the respective variable content section identifiers; and

transmit the respective variable content items to a first user computer for display on the test web page in variable content sections corresponding to the respective variable content section identifiers.

19. The system of claim 18, wherein the one or more computers are further operable to:

identify the second experiment of the one or more experiments associated with the page identifier, the second experiment being associated with the second set of variable content section identifiers;

for respective variable content section identifiers that have been received;

search for variable content associated with the second experiment and the respective variable content section identifiers;

if variable content associated with the second experiment and the respective variable content section identifiers is found;

select respective variable content items associated with the second experiment and the variable content section identifiers; and

transmit the respective variable content items to a second user computer for display on the test web page in variable content sections corresponding to the respective variable content section identifiers.

20. The system of claim 15, wherein the one or more computers are further operable to:

measure the first effectiveness of the test web page in achieving the goal, wherein the first effectiveness is based on the test web page displaying a first particular subset of variable content items associated with the first experiment and respective variable content section identifiers that have been received;

measure the second effectiveness of the test web page in achieving goal, wherein the second effectiveness is based on the test web page displaying a second particular subset of variable content items associated with the second experiment and respective variable content section identifiers that have been received; and

identify the most effective website content based on the first effectiveness and the second effectiveness.