Multi-campaign content allocation

• US 9,460,449 B2
• Filed: 05/16/2013
• Issued: 10/04/2016
• Est. Priority Date: 12/31/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Juda, Adam I., Baugher, Alexandra, Meyer, Michael M., Desikan, Pavan K., Mirkin, Ilia, Black, Julie A.
• Primary Examiner(s): Hamilton, Matthew L
• Application Number:
  US13/895,705
• Publication Number:
  US 20130254015A1
• Time in Patent Office: 1,237 Days
• Field of Search: 705/14
• US Class Current: 1/1
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0243 Comparative campaigns
• CPC Class Codes: G06Q 30/0244 Optimization

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for creating experiment content distribution campaigns (“experiment campaign”) and allocating content in accordance with the experiment campaign. In one aspect, a control campaign or an experiment campaign are selected as an available campaign for providing content in response to receiving a content request. The selection can be a probability constrained selection where the probability with which the experiment campaign is selected is specified as a portion of a total number of requests received during an experiment period for which the experiment campaign is available to provide content. Content is provided in response to the request, where the provided content being content corresponding to the selected campaign.

Claims

1. A method, comprising:

receiving, by one or more data processing apparatus, a request to create an experiment campaign based on an existing campaign;

receiving, by the one or more data processing apparatus through a communications network, data specifying an experiment parameter in the experiment campaign that differs from existing parameters of the existing campaign;

storing, by the one or more data processing apparatus and in computer memory, the experiment campaign, comprising storing a set of experiment difference data that i) excludes at least one of the existing parameters that is also a parameter of the experiment campaign, and ii) specifies a difference between the experiment parameter and the existing parameters;

receiving, by the one or more data processing apparatus through the communications network, a request to serve content to be presented on an electronic document requested by a user device executing an application at the user device, wherein the request includes a network address identifying the user device;

generating, after receiving the request for content by the one or more data processing apparatus, a set of experiment parameters for the experiment campaign using the at least one of the existing parameters from the existing campaign and the stored experiment difference data; and

in response to the request, transmitting, to the user device through the communications network, content that is integrated into a presentation of the electronic document by the user device, wherein the content is transmitted to the user device based on the set of experiment parameters.

2. The method of claim 1, comprising:

identifying, from among multiple different experiment campaigns that have been created from the existing campaign, one experiment campaign that will be used to provide content in response to the request, wherein;

generating a set of experiment parameters comprises generating a set of experiment parameters for the one experiment campaign based on a combination of the existing parameters and the experiment difference data; and

transmitting content comprises transmitting content based on the experiment parameters for the one experiment campaign.

3. The method of claim 2, wherein the identifying the one experiment campaign comprises pseudo-randomly selecting the one experiment campaign from among the multiple different experiment campaigns.

4. The method of claim 2, comprising:

determining, for each of two or more of the multiple different experiment campaigns, a performance measure specifying a user interaction rate for content provided using the experiment campaign; and

providing data that cause the performance measure for each of the two or more of the multiple different experiment campaigns and a performance measure for the existing campaign to be presented in a user interface.

5. The method of claim 4, comprising providing data that cause presentation of a variation identifier specifying that a difference between performance measures for the existing campaign and one of the multiple different experiment campaigns is greater than a pre-specified difference.

6. The method of claim 5, wherein the data cause the variation identifier to be presented next to the performance measure for the one of the multiple different experiment campaigns.

7. The method of claim 1, comprising:

receiving a start date for the experiment campaign specifying a date at which the experiment campaign becomes available for distributing content;

determining that the start date for the experiment campaign has been reached; and

activating the experiment campaign, wherein the generating and transmitting are performed after activating the experiment campaign.

8. A system comprising:

a data store storing existing parameters for an existing campaign; and

one or more data processing apparatus that interact with the data store and execute instructions that cause the one or more data processing apparatus to perform operations comprising;

receiving, through a communications network, a request to create an experiment campaign based on the existing campaign;

receiving data specifying an experiment parameter in the experiment campaign that differs from existing parameters of the existing campaign;

storing, in the data store, the experiment campaign, comprising storing a set of experiment difference data that i) excludes at least one of the existing parameters that is also a parameter of the experiment campaign, and ii) specifies a difference between the experiment parameter and the existing parameters;

receiving, through the communications network, a request to serve content to be presented on an electronic document requested by a user device executing an application at the user device, wherein the request includes a network address identifying the user device;

generating, after receiving the request for content, a set of experiment parameters for the experiment campaign using the at least one of the existing parameters from the existing campaign and the stored experiment difference data; and

in response to the request, transmitting, to the user device through the communications network, content that is integrated into a presentation of the electronic document by the user device, wherein the content is transmitted to the user device based on the set of experiment parameters.

9. The system of claim 8, wherein:

the instructions cause the one or more data processing apparatus to perform operations comprising identifying, from among multiple different experiment campaigns that have been created from the existing campaign, one experiment campaign that will be used to provide content in response to the request;

generating a set of experiment parameters comprises generating a set of experiment parameters for the one experiment campaign based on a combination of the existing parameters and the experiment difference data; and

transmitting content comprises transmitting content based on the experiment parameters for the one experiment campaign.

10. The system of claim 9, wherein the identifying the one experiment campaign comprises pseudo-randomly selecting the one experiment campaign from among the multiple different experiment campaigns.

11. The system of claim 9, wherein the instructions cause the one or more data processing apparatus to perform operations comprising:

determining, for each of two or more of the multiple different experiment campaigns, a performance measure specifying a user interaction rate for content provided using the experiment campaign; and

providing data that cause the performance measure for each of the two or more of the multiple different experiment campaigns and a performance measure for the existing campaign to be presented in a user interface.

12. The system of claim 11, wherein the instructions cause the one or more data processing apparatus to perform operations comprising providing data that cause presentation of a variation identifier specifying that a difference between performance measures for the existing campaign and one of the multiple different experiment campaigns is greater than a pre-specified difference.

13. The system of claim 12, wherein the data cause the variation identifier to be presented next to the performance measure for the one of the multiple different experiment campaigns.

14. The system of claim 8, wherein the instructions cause the one or more data processing apparatus to perform operations comprising:

receiving a start date for the experiment campaign specifying a date at which the experiment campaign becomes available for distributing content;

determining that the start date for the experiment campaign has been reached; and

activating the experiment campaign, wherein the generating and transmitting are performed after activating the experiment campaign.

15. A non-transitory computer readable medium storing instructions that upon execution of the instructions by one or more data processing apparatus cause the one or more data processing apparatus to perform operations comprising:

receiving, through a communications network, a request to create an experiment campaign based on an existing campaign;

receiving data specifying an experiment parameter in the experiment campaign that differs from existing parameters of the existing campaign;

storing, in a data store, the experiment campaign, comprising storing a set of experiment difference data that i) excludes at least one of the existing parameters that is also a parameter of the experiment campaign, and ii) specifies a difference between the experiment parameter and the existing parameters;

receiving, through the communications network, a request to serve content to be presented on an electronic document requested by a user device executing an application at the user device, wherein the request includes a network address identifying the user device;

generating, after receiving the request for content, a set of experiment parameters for the experiment campaign using the at least one of the existing parameters from the existing campaign and the stored experiment difference data; and

in response to the request, transmitting, to the user device through the communications network, content that is integrated into a presentation of the electronic document by the user device, wherein the content is transmitted to the user device based on the set of experiment parameters.

16. The non-transitory computer readable medium of claim 15, wherein:

the instructions cause the one or more data processing apparatus to perform operations comprising identifying, from among multiple different experiment campaigns that have been created from the existing campaign, one experiment campaign that will be used to provide content in response to the request;

generating a set of experiment parameters comprises generating a set of experiment parameters for the one experiment campaign based on a combination of the existing parameters and the experiment difference data; and

transmitting content comprises transmitting content based on the experiment parameters for the one experiment campaign.

17. The non-transitory computer readable medium of claim 16, wherein the identifying the one experiment campaign comprises pseudo-randomly selecting the one experiment campaign from among the multiple different experiment campaigns.

18. The non-transitory computer readable medium of claim 16, wherein the instructions cause the one or more data processing apparatus to perform operations comprising:

determining, for each of two or more of the multiple different experiment campaigns, a performance measure specifying a user interaction rate for content provided using the experiment campaign; and

providing data that cause the performance measure for each of the two or more of the multiple different experiment campaigns and a performance measure for the existing campaign to be presented in a user interface.

19. The non-transitory computer readable medium of claim 18, wherein the instructions cause the one or more data processing apparatus to perform operations comprising providing data that cause presentation of a variation identifier specifying that a difference between performance measures for the existing campaign and one of the multiple different experiment campaigns is greater than a pre-specified difference.

20. The non-transitory computer readable medium of claim 15, wherein the instructions cause the one or more data processing apparatus to perform operations comprising:

receiving a start date for the experiment campaign specifying a date at which the experiment campaign becomes available for distributing content;

determining that the start date for the experiment campaign has been reached; and

activating the experiment campaign, wherein the generating and transmitting are performed after activating the experiment campaign.