MULTI-CAMPAIGN CONTENT ALLOCATION

• US 20110161161A1
• Filed: 05/11/2010
• Published: 06/30/2011
• Est. Priority Date: 12/31/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Juda, Adam Isaac, Mirkin, Ilia, Black, Julie A., Baugher, Alexandra, Meyer, Michael M., Desikan, Pavan K., Bavor, Clayton W. JR.
• Granted Patent: US 8,554,619 B2
• Time in Patent Office: Days
• US Class Current: 705/14.42
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0243 Comparative campaigns
• CPC Class Codes: G06Q 30/0244 Optimization

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for creating experiment content distribution campaigns (“experiment campaign”) and allocating content in accordance with the experiment campaign. In one aspect, a control campaign or an experiment campaign are selected as an available campaign for providing content in response to receiving a content request. The selection can be a probability constrained selection where the probability with which the experiment campaign is selected is specified as a portion of a total number of requests received during an experiment period for which the experiment campaign is available to provide content. Content is provided in response to the request, where the provided content being content corresponding to the selected campaign.

Claims

1. A method performed by data processing apparatus, the method comprising:

receiving, by a data processing apparatus, a request to create an experiment campaign corresponding to a control campaign, the control campaign including campaign parameters with which allocation of content corresponding to the control campaign is controlled;

creating, by a data processing apparatus, an initial experiment campaign having initial parameters that match a set of campaign parameters for the control campaign;

receiving, by a data processing apparatus, experiment parameters that modify the initial experiment campaign to create the experiment campaign that corresponds to the control campaign, the experiment parameters being used to adjust at least one parameter from the initial parameters, the experiment parameters including data specifying an experiment period and a portion of content requests received during the experiment period for which the experiment campaign is available to provide content;

activating, by a data processing apparatus, the experiment campaign in response to a start condition, the start condition being a specified condition that upon detection causes the experiment campaign to be activated, wherein activation of the experiment campaign causes the experiment campaign to be available, as specified by the experiment parameters, to provide content in response to content requests;

computing, by a data processing apparatus and for content requests received during the experiment period, performance measures for the control campaign and the experiment campaign; and

providing, by a data processing apparatus, data that causes presentation of the performance measures corresponding to the control campaign and the experiment campaign, the data further causing presentation of variation identifiers that represent variations between performance measures for the control campaign and the experiment campaign that exceed a variation threshold.

2. The method of claim 1, further comprising:

receiving, over the experiment period, a plurality of requests for content to be presented on a user device;

for each of the requests;

selecting, as an available campaign for providing content in response to the request, a campaign from among the experiment campaign and the control campaign, the selection being based on the outcome of a probability constrained selection function, the probability constrained selection function being a function that selects the experiment campaign as the available campaign with a first probability and selects the control campaign as the available campaign with a second probability;

providing content in response to the request; and

storing performance measures for the available campaign.

3. The method of claim 2, further comprising computing the first probability and the second probability based on the portion of content requests received during the experiment period for which the experiment campaign is available to select content.

4. The method of claim 1, wherein receiving experiment parameters comprises receiving a percentage of all content requests received during the experiment period for which the experiment campaign is available to select content.

5. The method of claim 1, wherein receiving experiment parameters that modify the initial campaign to create the experiment campaign that corresponds to the control campaign, comprises receiving a bid for the experiment campaign that differs from a corresponding bid for the control campaign.

6. The method of claim 1, wherein receiving experiment parameters that modify the initial campaign to create the experiment campaign that corresponds to the control campaign, comprises receiving a targeting criterion for the experiment campaign that differs from existing targeting criteria for the control campaign.

7. The method of claim 1, wherein activating the experiment campaign in response to a start condition comprises activating the experiment campaign at a specified start time.

8. The method of claim 1, wherein activating the experiment campaign in response to a start condition comprises activating the experiment campaign in response to a user action requesting that the experiment campaign be activated.

9. The method of claim 1, wherein computing performance measures for the control campaign and the experiment campaign comprises:

computing an aggregate number of presentations of content corresponding to the experiment campaign;

computing an aggregate number of presentations of content corresponding to the control campaign;

computing an aggregate number of selections of content corresponding to the experiment campaign; and

computing an aggregate number of selections of content corresponding to the experiment campaign.

10. The method of claim 9, further comprising:

computing a variation between the performance measures for the control campaign and the performance measures for the experiment campaign; and

determining that the computed variation exceeds the variation threshold.

11. The method of claim 10, wherein determining that the computed variation exceeds a variation threshold comprises determining that the computed variation is statistically relevant based on the computed variation relative to a number of data points used to compute the variation.

12. The method of claim 1, further comprising storing, in response to receipt of the experiment parameters and in association with an experiment campaign identifier, experiment difference data specifying differences of the experiment parameters relative to the set of campaign parameters.

13. A system comprising:

a data store; and

an experiment subsystem coupled to the data store, the experiment subsystem comprising at least one processor configured to;

receive a request to create an experiment campaign corresponding to a control campaign, the control campaign including campaign parameters with which allocation of content corresponding to the control campaign is controlled;

create, in the data store, an initial experiment campaign having initial parameters that match a set of campaign parameters for the control campaign;

receive experiment parameters that modify the initial experiment campaign to create, in the data store, the experiment campaign that corresponds to the control campaign, the experiment parameters being used to adjust at least one parameter from the initial parameters, the experiment parameters including data specifying an experiment period and a portion of content requests received during the experiment period for which the experiment campaign is available to provide content;

activate the experiment campaign in response to a start condition, the start condition being a specified condition that upon detection causes the experiment campaign to be activated, wherein activation of the experiment campaign causes the experiment campaign to be available, as specified by the experiment parameters, to provide content in response to content requests;

compute, for content requests received during the experiment period, performance measures for the control campaign and the experiment campaign, the computed performance measures being stored in the data store; and

provide data that causes presentation of the performance measures corresponding to the control campaign and the experiment campaign, the data further causing presentation of variation identifiers that represent variations between performance measures for the control campaign and the experiment campaign that exceed a variation threshold.

14. The system of claim 13, wherein the experiment subsystem is further configured to:

receive, over the experiment period, a plurality of requests for content to be presented on a user device;

for each of the requests;

select, as an available campaign for providing content in response to the request, a campaign from among the experiment campaign and the control campaign, the selection being based on the outcome of a probability constrained selection function, the probability constrained selection function being a function that selects the experiment campaign as the available campaign with a first probability and selects the control campaign as the available campaign with a second probability;

provide content in response to the request; and

store, in the data store, performance measures for the available campaign.

15. The system of claim 13, wherein the experiment subsystem is further configured to compute the first probability and the second probability based on the portion of content requests received during the experiment period for which the experiment campaign is available to select content.

16. The system of claim 13, wherein the experiment subsystem is further configured to receive experiment parameters specifying a percentage of all content requests received during the experiment period for which the experiment campaign is available to select content.

17. The system of claim 13, wherein the experiment subsystem is further configured to receive experiment parameters that specify a bid for the experiment campaign that differs from a corresponding bid for the control campaign.

18. The system of claim 13, wherein the experiment subsystem is further configured to receive experiment parameters that specify a targeting criterion for the experiment campaign that differs from existing targeting criteria for the control campaign.

19. The system of claim 13, wherein the experiment subsystem is further configured to activate the experiment campaign at a specified start time.

20. The system of claim 13, wherein the experiment subsystem is further configured to activate the experiment campaign in response to a user action requesting that the experiment campaign be activated.

21. The system of claim 13, wherein the experiment subsystem is further configured to:

compute an aggregate number of presentations of content corresponding to the experiment campaign;

compute an aggregate number of presentations of content corresponding to the control campaign;

compute an aggregate number of selections of content corresponding to the experiment campaign; and

compute an aggregate number of selections of content corresponding to the experiment campaign.

22. The system of claim 21, wherein the experiment subsystem is further configured to:

compute a variation between the performance measures for the control campaign and the performance measures for the experiment campaign; and

determine that the computed variation exceeds the variation threshold.

23. The system of claim 22, wherein the experiment subsystem is further configured to determine that the computed variation is statistically relevant based on the computed variation relative to a number of data points used to compute the variation.

24. The system of claim 22, wherein the experiment subsystem is further configured to store in the data store in association with an experiment campaign identifier, experiment difference data specifying differences of the experiment parameters relative to the set of campaign parameters.