Multi-campaign content allocation

• US 8,554,619 B2
• Filed: 05/11/2010
• Issued: 10/08/2013
• Est. Priority Date: 12/31/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Juda, Adam I., Baugher, Alexandra, Meyer, Michael M., Desikan, Pavan K., Mirkin, Ilia, Black, Julie A.
• Primary Examiner(s): Hamilton, Matthew L
• Application Number:
  US12/777,593
• Publication Number:
  US 20110161161A1
• Time in Patent Office: 1,246 Days
• Field of Search: 705/14, 705/14.4
• US Class Current: 705/14.4
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0243 Comparative campaigns
• CPC Class Codes: G06Q 30/0244 Optimization

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for creating experiment content distribution campaigns (“experiment campaign”) and allocating content in accordance with the experiment campaign. In one aspect, a control campaign or an experiment campaign are selected as an available campaign for providing content in response to receiving a content request. The selection can be a probability constrained selection where the probability with which the experiment campaign is selected is specified as a portion of a total number of requests received during an experiment period for which the experiment campaign is available to provide content. Content is provided in response to the request, where the provided content being content corresponding to the selected campaign.

Claims

1. A method performed by data processing apparatus, the method comprising:

creating, by a data processing apparatus, an initial experiment campaign having initial parameters that match a set of campaign parameters for a control campaign;

receiving, by a data processing apparatus, from an advertiser computer, experiment parameters that modify the initial experiment campaign to create an experiment campaign that differs from the control campaign;

generating, by a data processing apparatus, experiment difference data associated with the experiment campaign based on differences between the experiment parameters for the experiment campaign and the set of campaign parameters for the control campaign, the generated experiment difference data excluding one or more experiment parameters that match a control campaign parameter in the set of campaign parameters;

receiving, by a data processing apparatus, a request for content from a user device; and

providing, in response to the request and for presentation at the user device, content from the experiment campaign, the content being provided based on both the control campaign parameters that match the one or more excluded experiment parameters and the experiment difference data associated with the experiment campaign.

2. The method of claim 1, further comprising:

receiving, over an experiment period, a plurality of requests for content to be presented on a user device;

for each of the requests;

selecting, as an available campaign for providing content in response to the request, a campaign from among the experiment campaign and the control campaign, the selection being based on an outcome of a probability constrained selection function, the probability constrained selection function being a function that selects the experiment campaign as the available campaign with a first probability and selects the control campaign as the available campaign with a second probability;

providing content in response to the request; and

storing performance measures for the available campaign; and

providing data that causes presentation of performance measures corresponding to the control campaign and performance measures corresponding to the experiment campaign, the data further causing presentation of variation identifiers that represent variations between performance measures for the control campaign and the experiment campaign that exceed a variation threshold.

3. The method of claim 2, further comprising computing the first probability and the second probability based on a portion of content requests received during the experiment period for which the experiment campaign is available to select content.

4. The method of claim 1, wherein the receiving experiment parameters comprises receiving a percentage of all content requests received during an experiment period for which the experiment campaign is available to select content.

5. The method of claim 1, wherein the receiving experiment parameters that modify the initial experiment campaign to create the experiment campaign that corresponds to the control campaign, comprises receiving a bid for the experiment campaign that differs from a corresponding bid for the control campaign.

6. The method of claim 1, wherein the receiving experiment parameters that modify the initial experiment campaign to create the experiment campaign that corresponds to the control campaign, comprises receiving a targeting criterion for the experiment campaign that differs from existing targeting criteria for the control campaign.

7. The method of claim 1, further comprising activating the experiment campaign at a specified start time.

8. The method of claim 1, further comprising activating the experiment campaign in response to an action requesting that the experiment campaign be activated.

9. The method of claim 2, further comprising:

computing an aggregate number of presentations of content corresponding to the experiment campaign;

computing an aggregate number of presentations of content corresponding to the control campaign;

computing an aggregate number of selections of content corresponding to the experiment campaign; and

computing an aggregate number of selections of content corresponding to the experiment campaign.

10. The method of claim 9, further comprising:

computing a variation between the performance measures for the control campaign and the performance measures for the experiment campaign; and

determining that the computed variation exceeds a variation threshold.

11. The method of claim 10, wherein the determining that the computed variation exceeds a variation threshold comprises determining that the computed variation is statistically relevant based on the computed variation relative to a number of data points used to compute the variation.

12. The method of claim 1, further comprising storing, in response to receipt of the experiment parameters and in association with an experiment campaign identifier, the experiment difference data specifying differences of the experiment parameters relative to the campaign parameters.

13. A system comprising:

a data store; and

an experiment subsystem coupled to the data store, the experiment subsystem comprising at least one processor configured to;

create, in the data store, an initial experiment campaign having initial parameters that match a set of campaign parameters for control campaign;

receive, from an advertiser computer, experiment parameters that modify the initial experiment campaign to create, in the data store, an experiment campaign that differs from the control campaign;

generate experiment difference data associated with the experiment campaign based on differences between the experiment parameters for the experiment campaign and the set of campaign parameters for the control campaign, the generated experiment difference data excluding one or more experiment parameters that match a control campaign parameter in the set of campaign parameters;

receive a request for content from a user device; and

provide, in response to the request and for presentation at the user device, content from the experiment campaign, the content being provided based on both the control campaign parameters that match the one or more excluded experiment parameters and the experiment difference data associated with the experiment campaign.

14. The system of claim 13, wherein the experiment subsystem is further configured to:

receive, over an experiment period, a plurality of requests for content to be presented on a user device;

for each of the requests;

select, as an available campaign for providing content in response to the request, a campaign from among the experiment campaign and the control campaign, the selection being based on an outcome of a probability constrained selection function, the probability constrained selection function being a function that selects the experiment campaign as the available campaign with a first probability and selects the control campaign as the available campaign with a second probability;

provide content in response to the request; and

store, in the data store, performance measures for the available campaign; and

provide data that causes presentation of the performance measures corresponding to the control campaign and the experiment campaign, the data further causing presentation of variation identifiers that represent variations between performance measures for the control campaign and the experiment campaign that exceed a variation threshold.

15. The system of claim 14, wherein the experiment subsystem is further configured to compute the first probability and the second probability based on a portion of content requests received during the experiment period for which the experiment campaign is available to select content.

16. The system of claim 13, wherein the experiment subsystem is further configured to receive experiment parameters specifying a percentage of all content requests received during an experiment period for which the experiment campaign is available to select content.

17. The system of claim 13, wherein the experiment subsystem is further configured to receive experiment parameters that specify a bid for the experiment campaign that differs from a corresponding bid for the control campaign.

18. The system of claim 13, wherein the experiment subsystem is further configured to receive experiment parameters that specify a targeting criterion for the experiment campaign that differs from existing targeting criteria for the control campaign.

19. The system of claim 13, wherein the experiment subsystem is further configured to activate the experiment campaign at a specified start time.

20. The system of claim 13, wherein the experiment subsystem is further configured to activate the experiment campaign in response to an action requesting that the experiment campaign be activated.

21. The system of claim 14, wherein the experiment subsystem is further configured to:

compute an aggregate number of presentations of content corresponding to the experiment campaign;

compute an aggregate number of presentations of content corresponding to the control campaign;

compute an aggregate number of selections of content corresponding to the experiment campaign; and

compute an aggregate number of selections of content corresponding to the experiment campaign.

22. The system of claim 21, wherein the experiment subsystem is further configured to:

compute a variation between the performance measures for the control campaign and the performance measures for the experiment campaign; and

determine that the computed variation exceeds a variation threshold.

23. The system of claim 22, wherein the experiment subsystem is further configured to determine that the computed variation is statistically relevant based on the computed variation relative to a number of data points used to compute the variation.

24. The system of claim 22, wherein the experiment subsystem is further configured to store in the data store in association with an experiment campaign identifier, the experiment difference data specifying differences of the experiment parameters relative to the campaign parameters.