System and Method for Improving the Performance of Electronic Media Advertising Campaigns Through Multi-Attribute Analysis and Optimization

• US 20110238487A1
• Filed: 05/03/2011
• Published: 09/29/2011
• Est. Priority Date: 12/15/2000
• Status: Abandoned Application

• Current Assignee: Google LLC (Alphabet Inc.), Paramark, Inc. (Newworldiq, Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Weisinger, James R., Kumar, Sachin, Ranka, Sanjay, Chang, E. Diane, Chatwin, Richard E., Lenderman, Jason
• Application Number:
  US13/099,608
• Publication Number:
  US 20110238487A1
• Time in Patent Office: Days
• US Class Current: 705/14.42
• CPC Class Codes: G06Q 10/06375 Prediction of business process outcome or impact based on a proposed change
• CPC Class Codes: G06Q 10/06393 Score-carding, benchmarking or key performance indicator [KPI] analysis
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0203 Market surveys; Market polls
• CPC Class Codes: G06Q 30/0204 Market segmentation
• CPC Class Codes: G06Q 30/0207 Discounts or incentives, e.g. coupons or rebates
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06Q 30/0242 Determining effectiveness of advertisements
• CPC Class Codes: G06Q 30/0243 Comparative campaigns
• CPC Class Codes: G06Q 30/0244 Optimization
• CPC Class Codes: G06Q 30/0277 Online advertisement

Abstract

Automated system, methods, algorithms, procedures, and computer software programs and computer program products for improving and optimizing the performance of messaging campaigns, particularly for marketing campaigns in which advertisements or other messages are distributed over an interactive measurable medium such as the Internet. Analysis and Optimization method and procedure, an automated system, and system and method that exploit the underlying multi-attribute structure, as well as other features and advantages. Optimization procedures allocate the ad alternatives or other message to the customer population to optimize business objectives such as maximizing the number of positive responses received. Procedure for generating message allocations that improve and attempt to optimize the campaign performance. Methods ensure that campaign constraints are not violated. Methods can be implemented on a computer that is programmed to retrieve message performance information and to generate recommended message allocations for each stage in a multi-stage messaging campaign to achieve messaging goals.

Claims

1-22. -22. (canceled)

23. A computer-implemented method, comprising:

accessing a multi-dimensional index of message attributes that is defined by a set of matching attributes for a plurality of message alternatives belonging to the multi-dimensional index, with the matching attributes being attributes that define content of the plurality of message alternatives, and each message alternative belonging to the multi-dimensional index having attribute values specified for the matching attributes;

receiving performance data for the plurality of message alternatives;

computing importance measures for the attribute values corresponding to the matching attributes, each importance measure for each attribute value being computed based on the performance data and specifying a relative measure of importance of the attribute value for predicting message alternative performance;

selecting, from available message alternatives, a message alternative for presentation, the message alternative being based on the importance measures; and

providing the selected message alternative for presentation during a next stage of a message campaign.

24. The method of claim 23, further comprising generating the multi-dimensional index based on matching attributes for the plurality of message alternatives.

25. The method of claim 24, wherein generating the multi-dimensional index comprises:

selecting two or more attributes as the set of matching attributes that define the multi-dimensional index;

selecting message alternatives having attribute values corresponding to each attribute in the set of matching attributes; and

indexing each message alternative at a location of the multi-dimensional index corresponding to the attribute values for the message alternative.

26. The method of claim 23, wherein receiving performance data for the plurality of message alternatives comprises receiving, for each message alternative in a message campaign, data representing a performance metric for the message alternative over one or more previous stages of the message campaign.

27. The method of claim 26, wherein receiving data representing a performance metric comprises receiving data representing a presentation of the message alternative and a selection of the message alternative in response to the presentation of the message alternative.

28. The method of claim 26, wherein each previous stage of the message campaign is defined by a measure of time by which performance data is delineated.

29. The method of claim 28, wherein the performance data for each previous stage is weighted based on an elapsed time since an occurrence of actions represented by the performance data.

30. The method of claim 23, wherein computing importance measures for the attribute values corresponding to the matching attributes comprises computing, for combinations of attribute values corresponding to the matching attributes, aggregate performance measures for message alternatives having the combinations of attribute values.

31. The method of claim 30, wherein computing the aggregate performance measure comprises computing likelihoods that message alternatives having the combinations of attribute values are selected by a user, the likelihoods being computed relative to a likelihood that a baseline message alternative having baseline attribute values is selected by a user.

32. The method of claim 30, wherein selecting a message alternative for presentation comprises:

identifying available message alternatives;

accessing, for each available message alternative, an aggregate performance measure for message alternatives having a same combination of attribute values as the available message alternative; and

selecting, for presentation during a next stage of the message campaign, the available message alternative having at least a threshold aggregate performance measure.

33. The method of claim 32, wherein selecting the available message alternative having at least a threshold aggregate performance measure comprises selecting the available message alternative having a highest aggregate performance measure.

34. A system comprising:

a memory device storing a multi-dimensional index of message attributes that is defined by a set of matching attributes for a plurality of message alternatives belonging to the multi-dimensional index, with the matching attributes being attributes that define content of the plurality of message alternatives, and each message alternative belonging to the multi-dimensional index having attribute values specified for the matching attributes; and

one or more processors configured to interact with memory device, the one or more processors being further configured to perform operations including;

accessing the multi-dimensional index;

receiving performance data for the plurality of message alternatives;

computing importance measures for the attribute values corresponding to the matching attributes, each importance measure, for each attribute value, being computed based on the performance data and specifying a relative measure of importance of the attribute value for predicting message alternative performance;

selecting, from available message alternatives, a message alternative for presentation, the message alternative being based on the importance measures; and

providing the selected message alternative for presentation during a next stage of a message campaign.

35. The system of claim 34, wherein the one or more processors are further configured to perform operations including:

generating the multi-dimensional index based on matching attributes for the plurality of message alternatives; and

storing the multi-dimensional index in the memory device.

36. The system of claim 35, wherein the one or more processors are further configured to perform operations including:

selecting two or more attributes as the set of matching attributes that define the multi-dimensional index;

selecting message alternatives having attribute values corresponding to each attribute in the set of matching attributes; and

indexing each message alternative at a location of the multi-dimensional index corresponding to the attribute values for the message alternative.

37. The system of claim 34, wherein the one or more processors are further configured to perform operations including receiving, for each message alternative in a message campaign, data representing a performance metric for the message alternative over one or more previous stages of the message campaign.

38. The system of claim 37, wherein the one or more processors are further configured to perform operations including receiving data representing a presentation of the message alternative and a selection of the message alternative in response to the presentation of the message alternative.

39. The system of claim 37, wherein each previous stage of the message campaign is defined by a measure of time by which performance data is delineated.

40. The system of claim 39, wherein the performance data for each previous stage is weighted based on an elapsed time since an occurrence of actions represented by the performance data.

41. The system of claim 37, wherein the one or more processors are further configured to perform operations including computing, for combinations of attribute values corresponding to the matching attributes, aggregate performance measures for message alternatives having the combinations of attribute values.

42. The system of claim 41, wherein the one or more processors are further configured to perform operations including computing likelihoods that message alternatives having the combinations of attribute values are selected by a user, the likelihoods being computed relative to a likelihood that a baseline message alternative having baseline attribute values is selected by a user.

43. The system of claim 41, wherein the one or more processors are further configured to perform operations including:

identifying available message alternatives;

accessing, for each available message alternative, an aggregate performance measure for message alternatives having a same combination of attribute values as the available message alternative; and

selecting, for presentation during a next stage of the message campaign, the available message alternative having at least a threshold aggregate performance measure.

44. The system of claim 43, wherein the one or more processors are further configured to perform operations including selecting the available message alternative having a highest aggregate performance measure.

45. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

accessing a multi-dimensional index of message attributes that is defined by a set of matching attributes for a plurality of message alternatives belonging to the multi-dimensional index, with the matching attributes being attributes that define content of the plurality of message alternatives, and each message alternative belonging to the multi-dimensional index having attribute values specified for the matching attributes;

receiving performance data for the plurality of message alternatives;

computing importance measures for the attribute values corresponding to the matching attributes, each importance measure, for each attribute value, being computed based on the performance data and specifying a relative measure of importance of the attribute value for predicting message alternative performance;

selecting, from available message alternatives, a message alternative for presentation, the message alternative being based on the importance measures; and

providing the selected message alternative for presentation during a next stage of a message campaign.

46. A computer-implemented method, comprising:

receiving a set of attributes and candidate attribute values;

accessing a multi-dimensional index of advertisement attributes that is defined by the set of attributes and performance data for advertisements having attribute values for each attribute in the set of attributes, with each attribute in the set of attributes being an attribute that defines content of one of the advertisements, and the performance data representing performance measures for the advertisements over one or more advertising stages;

computing, based on the multi-dimensional index, a performance estimate for an advertisement having candidate attribute values for the set of attributes; and

providing a report including the performance estimate.

47. The method of claim 46, further comprising:

computing, based on the multi-dimensional index, additional performance estimates for advertisements having other attribute values for the set of attributes, wherein the report includes the additional performance estimates.

48. The method of claim 47, wherein the additional performance estimates include an indication of other attribute values for the set of attributes that correspond to a highest performance estimate.

49. The method of claim 46, further comprising generating the multi-dimensional index based on matching attributes for the advertisements.

50. The method of claim 49, wherein generating the multi-dimensional index comprises:

selecting two or more attributes as the set of matching attributes that define the multi-dimensional index;

selecting the advertisements that have attribute values corresponding to each attribute in the set of matching attributes; and

indexing each selected advertisement at a location of the multi-dimensional index corresponding to the attribute values for advertisement.

51. The method of claim 46, wherein computing a performance estimate comprises computing a value representing an estimated ratio of a number of selections of the advertisement in response to presentation of the advertisement relative to a number of presentations of the advertisement.

52. The method of claim 46, wherein each advertising stage is defined by a measure of time by which performance data is delineated.

53. The method of claim 52, wherein the performance data for each advertising stage is weighted based on an elapsed time since an occurrence of actions represented by the performance data.

54. The method of claim 46, wherein computing the performance estimate comprises computing a likelihood that the advertisement having the candidate attribute values will be selected by a user, the likelihoods being computed relative to a likelihood that a baseline advertisement having baseline attribute values is selected by a user.

55. A system comprising:

a memory device storing a multi-dimensional index of advertisement attributes that is defined by a set of attributes and performance data for advertisements having attribute values for each attribute in the set of attributes, with each attribute in the set of attributes being an attribute that defines content of one of the advertisements, and the performance data representing performance measures for the advertisements over one or more advertising stages; and

one or more processors configured to interact with the memory device, the one or more processor further configured to perform operations including;

receiving a set of attributes and candidate attribute values;

accessing the multi-dimensional index in response to receiving the set of attributes and candidate attribute values;

computing, based on the multi-dimensional index, a performance estimate for an advertisement having candidate attribute values for the set of attributes; and

providing a report including the performance estimate.

56. The system of claim 55, wherein the one or more processors are further configured to perform operations including computing, based on the multi-dimensional index, additional performance estimates for advertisements having other attribute values for the set of attributes, wherein the report includes the additional performance estimates.

57. The system of claim 56, wherein the additional performance estimates include an indication of other attribute values for the set of attributes that correspond to a highest performance estimate.

58. The system of claim 55, wherein the one or more processors are further configured to perform operations including generating the multi-dimensional index based on matching attributes for the advertisements.

59. The system of claim 58, wherein the one or more processors are further configured to perform operations including:

selecting two or more attributes as the set of matching attributes that define the multi-dimensional index;

selecting the advertisements that have attribute values corresponding to each attribute in the set of matching attributes; and

indexing each selected advertisement at a location of the multi-dimensional index corresponding to the attribute values for the advertisement.

60. The system of claim 55, wherein the one or more processors are further configured to perform operations including computing a value representing an estimated ratio of a number of selections of the advertisement in response to presentation of the advertisement relative to a number of presentations of the advertisement.

61. The system of claim 55, wherein each advertising stage is defined by a measure of time by which performance data is delineated.

62. The system of claim 61, wherein the performance data for each advertising stage is weighted based on an elapsed time since an occurrence of actions represented by the performance data.

63. The system of claim 55, wherein the one or more computers are further configured to perform operations including computing a likelihood that the advertisement having the candidate attribute values will be selected by a user, the likelihoods being computed relative to a likelihood that a baseline advertisement having baseline attribute values is selected by a user.

64. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

receiving a set of attributes and candidate attribute values;

accessing a multi-dimensional index of advertisement attributes that is defined by the set of attributes and performance data for advertisements having attribute values for each attribute in the set of attributes, with each attribute in the set of attributes being an attribute that defines content of one of the advertisements, and the performance data representing performance measures for the advertisements over one or more advertising stages;

computing, based on the multi-dimensional index, a performance estimate for an advertisement having candidate attribute values for the set of attributes; and

providing a report including the performance estimate.

65. A computer-implemented method, comprising:

receiving performance data for advertisements that have been presented to users from a plurality of market segments;

for each market segment;

generating a multi-dimensional index of advertisement attributes that is defined by a unique set of matching attributes for the advertisements that have been presented to users in the market segment, with the matching attributes being attributes that define content of the advertisements, and each advertisement belonging to the multi-dimensional index having attribute values specified for the matching attributes; and

computing importance measures for combinations of attribute values for the matching attributes, with each importance measure representing an effect of an attribute value on advertisement performance in the market segment; and

providing a report including the importance measures for the combinations of attribute values.

66. The method of claim 65, wherein providing a report comprises providing a report including suggested values for the matching attributes for advertisements being targeted to specified market segments, the suggested values for the matching attributes being suggested values having an importance measure that satisfies a threshold importance measure.

67. The method of claim 65, wherein each market segment is defined based on user profile data for users to which the advertisements have been presented during one or more previous stages of an advertising campaign.

68. The method of claim 65, wherein generating the multi-dimensional index comprises:

selecting two or more attributes as the set of matching attributes that define the multi-dimensional index;

selecting advertisements having attribute values corresponding to each attribute in the set of matching attributes; and

indexing each advertisement at a location of the multi-dimensional index corresponding to the attribute values for the advertisement.

69. The method of claim 65, wherein receiving performance data for advertisements further comprises receiving, for each advertisement in a message campaign, data representing a performance metric for the advertisement over one or more previous stages of the message campaign.

70. The method of claim 69, wherein receiving data representing a performance metric further comprises receiving data representing each of a presentation of the advertisement and a selection of the advertisement in response to the presentation of the advertisement.

71. The method of claim 69, wherein each previous stage of the message campaign is defined by a measure of time by which performance data is delineated.

72. The method of claim 71, wherein the performance data for each previous stage is weighted based on an elapsed time since an occurrence of actions represented by the performance data.

73. The method of claim 65, wherein computing importance measures for combinations of attribute values for the matching attributes comprises computing, for the combinations of attribute values for the matching attributes, aggregate performance measures for advertisements having the combinations of attribute values.

74. The method of claim 73, wherein computing the aggregate performance measure comprises computing likelihoods that advertisements having the combinations of attribute values will be selected by a user, the likelihoods being computed relative to a likelihood that a baseline advertisement having baseline attribute values is selected by a user.

75. A system comprising:

a memory device storing performance data for advertisements that have been presented to users from a plurality of market segments; and

one or more computers configured to interact with the memory device, the one or more computers being further configured to perform operations including;

receiving, from the memory device the performance data;

for each market segment;

generating a multi-dimensional index of advertisement attributes that is defined by a unique set of matching attributes for the advertisements that have been presented to users in the market segment, with the matching attributes being attributes that define content of the advertisements, and each advertisement belonging to the multi-dimensional index having attribute values specified for the matching attributes; and

computing importance measures for combinations of attribute values for the matching attributes, with each importance measure representing an effect of an attribute value on advertisement performance in the market segment; and

providing a report including the importance measures for the combinations of attribute values.

76. The system of claim 75, wherein the one or more processors are further configured to perform operations including providing a report including suggested values for the matching attributes for advertisements being targeted to specified market segments, the suggested values for the matching attributes being suggested values having an importance measure that satisfies a threshold importance measure.

77. The system of claim 76, wherein each market segment is defined based on user profile data for users to which the advertisements have been presented during one or more previous stages of an advertising campaign.

78. The system of claim 75, wherein the one or more processors are further configured to perform operations including:

selecting two or more attributes as the set of matching attributes that define the multi-dimensional index;

selecting advertisements having attribute values corresponding to each attribute in the set of matching attributes; and

indexing each advertisement at a location of the multi-dimensional index corresponding to the attribute values for the advertisement.

79. The system of claim 75, wherein the one or more processors are further configured to perform operations including receiving, for each advertisement in a message campaign, data representing a performance metric for the advertisement over one or more previous stages of the message campaign.

80. The system of claim 79, wherein the one or more processors are further configured to perform operations including receiving data representing each of a presentation of the advertisement and a selection of the advertisement in response to the presentation of the advertisement.

81. The system of claim 79, wherein each previous stage of the message campaign is defined by a measure of time by which performance data is delineated.

82. The system of claim 79, wherein the performance data for each previous stage is weighted based on an elapsed time since an occurrence of actions represented by the performance data.

83. The system of claim 75, wherein the one or more processors are further configured to perform operations including computing, for the combinations of attribute values for the matching attributes, aggregate performance measures for advertisements having the combinations of attribute values.

84. The system of claim 83, wherein the one or more processors are further configured to perform operations including computing likelihoods that advertisements having the combinations of attribute values will be selected by a user, the likelihoods being computed relative to a likelihood that a baseline advertisement having baseline attribute values is selected by a user.

85. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

receiving performance data for advertisements that have been presented to users from a plurality of market segments;

for each market segment;

generating a multi-dimensional index of advertisement attributes that is defined by a unique set of matching attributes for the advertisements that have been presented to users in the market segment, with the matching attributes being attributes that define content of the advertisements, and each advertisement belonging to the multi-dimensional index having attribute values specified for the matching attributes; and

computing importance measures for combinations of attribute values for the matching attributes, with each importance measure representing an effect of an attribute value on advertisement performance in the market segment; and

providing a report including the importance measures for the combinations of attribute values.

86. A computer-implemented method, comprising

receiving performance data for a plurality of advertisements that have been presented to users from a plurality of market segments;

for each market segment;

generating a multi-dimensional index of advertisement attributes that is defined by a unique set of matching attributes for the advertisements that have been presented to users in the market segment, with the matching attributes being attributes that define content of the advertisements, and each advertisement belonging to the multi-dimensional index having attribute values specified for the matching attributes; and

computing an importance measure for each attribute value corresponding to the matching attributes, each importance measure, for each attribute value, specifying a relative measure of importance of the attribute value for predicting message alternative performance in the market segment;

receiving targeting data specifying a target market segment for an advertisement; and

providing a report specifying recommended attribute values for the advertisement, the recommended attribute values being computed based on the targeting data and the multi-dimensional index and specifying attribute values having at least a threshold importance measure for the target market.

87. The method of claim 86, wherein generating the multi-dimensional index comprises:

selecting two or more attributes as the set of matching attributes that define the multi-dimensional index;

selecting advertisements having attribute values corresponding to each attribute in the set of matching attributes; and

indexing each advertisements at a location of the multi-dimensional index corresponding to the attribute values for the advertisement.

88. The method of claim 86, wherein receiving performance data for the plurality of advertisements comprises receiving, for each message alternative in a message campaign, data representing a performance metric for the message alternative over one or more previous stages of the message campaign.

89. The method of claim 88, wherein receiving performance data comprises receiving data representing a number of presentations of each advertisement and a number of selections of each advertisement in response to the presentation of the advertisement.

90. The method of claim 88, wherein each previous stage of the message campaign is defined by a measure of time by which performance data is delineated.

91. The method of claim 90, wherein the performance data for each previous stage is weighted based on an elapsed time since an occurrence of actions represented by the performance data.

92. The method of claim 86, wherein computing an importance measure for each attribute value corresponding to the matching attributes comprises computing, for combinations of attribute values corresponding to the matching attributes, aggregate performance measures for advertisements having the combinations of attribute values.

93. The method of claim 92, wherein computing the aggregate performance measure comprises computing likelihoods that advertisements having the combinations of attribute values will be selected by a user, the likelihoods being computed relative to a likelihood that a baseline advertisement having baseline attribute values is selected by a user.

94. A system comprising:

a memory device storing performance data for a plurality of advertisements that have been presented to users from a plurality of market segments; and

one or more processors configured to interact with the memory device, the one or more processors being further configured to perform operations including;

receiving the performance data from the memory device;

generating a multi-dimensional index of advertisement attributes that is defined by a unique set of matching attributes for the advertisements that have been presented to users in the market segment, with the matching attributes being attributes that define content of the advertisements, and each advertisement belonging to the multi-dimensional index having attribute values specified for the matching attributes; and

computing an importance measure for each attribute value corresponding to the matching attributes, each importance measure, for each attribute value, specifying a relative measure of importance of the attribute value for predicting message alternative performance in the market segment;

receiving targeting data specifying a target market segment for an advertisement; and

providing a report specifying recommended attribute values for the advertisement, the recommended attribute values being computed based on the targeting data and the multi-dimensional index and specifying attribute values having at least a threshold importance measure for the target market.

95. The system of claim 94, wherein the one or more processors are further configured to perform operations including:

selecting two or more attributes as the set of matching attributes that define the multi-dimensional index;

selecting advertisements having attribute values corresponding to each attribute in the set of matching attributes; and

indexing each advertisements at a location of the multi-dimensional index corresponding to the attribute values for the advertisement.

96. The system of claim 94, wherein the one or more processors are further configured to perform operations including receiving, for each message alternative in a message campaign, data representing a performance metric for the message alternative over one or more previous stages of the message campaign.

97. The system of claim 96, wherein receiving performance data comprises receiving data representing a number of presentations of each advertisement and a number of selections of each advertisement in response to the presentation of the advertisement.

98. The system of claim 96, wherein each previous stage of the message campaign is defined by a measure of time by which performance data is delineated.

99. The system of claim 98, wherein the performance data for each previous stage is weighted based on an elapsed time since an occurrence of actions represented by the performance data.

100. The system of claim 94, wherein the one or more processors are further configured to perform operations including computing, for combinations of attribute values corresponding to the matching attributes, aggregate performance measures for advertisements having the combinations of attribute values.

101. The system of claim 100, wherein the one or more processors are further configured to perform operations including computing likelihoods that advertisements having the combinations of attribute values will be selected by a user, the likelihoods being computed relative to a likelihood that a baseline advertisement having baseline attribute values is selected by a user.

102. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

receiving performance data for a plurality of advertisements that have been presented to users from a plurality of market segments;

for each market segment;

generating a multi-dimensional index of advertisement attributes that is defined by a unique set of matching attributes for the advertisements that have been presented to users in the market segment, with the matching attributes being attributes that define content of the advertisements, and each advertisement belonging to the multi-dimensional index having attribute values specified for the matching attributes; and

computing an importance measure for each attribute value corresponding to the matching attributes, each importance measure specifying a relative measure of importance of the attribute value for predicting message alternative performance in the market segment;

receiving targeting data specifying a target market segment for an advertisement; and

providing a report specifying recommended attribute values for the advertisement, the recommended attribute values being computed based on the targeting data and the multi-dimensional index and specifying attribute values having at least a threshold importance measure for the target market.

103. A computer-implemented method, comprising:

allocating impressions to each advertisement in a set of advertisements over stages of an advertising campaign;

receiving performance data representing performance of each advertisement over the stages of the advertising campaign;

assigning each advertisement to a multi-dimensional index of advertisement attributes that is defined by a set of attributes for which the advertisement has corresponding attribute values, with each attribute in the set of attributes being an attribute that defines content of the advertisement, and each advertisement being assigned to only one multi-dimensional index;

computing performance measures for combinations of attribute values from the set of attributes, with each performance measure being computed based on the performance data of advertisements assigned to the multi-dimensional index and having the combinations of attribute values for which the performance measure is being computed; and

allocating impressions for a future stage of the advertising campaign to the each advertisement in the set of advertisements based on a function of the performance measures corresponding to the combinations of attribute values for the advertisements.

104. The method of claim 103, wherein receiving performance data comprises receiving data representing a number of presentations of each advertisement and a number of selections of the advertisement in response to the presentation of the advertisement.

105. The method of claim 103, wherein each of the stages of the advertisement campaign is defined by a measure of time by which performance data is delineated.

106. The method of claim 105, wherein the performance data for each previous stage is weighted based on an elapsed time since an occurrence of actions represented by the performance data.

107. A system comprising:

a memory device storing a multi-dimensional index of advertisement attributes that is defined by a set of attributes for which advertisements have corresponding attribute values, with each attribute in the set of attributes being an attribute that defines content of an advertisement; and

one or more processors configured to interact with the memory device, the one or more processors being further configured to perform operations comprising;

allocating impressions to each advertisement in a set of advertisements over stages of an advertising campaign;

receiving performance data representing performance of each advertisement over the stages of the advertising campaign;

assigning each advertisement in the set to only one multi-dimensional index location;

computing performance measures for combinations of attribute values from the set of attributes, with each performance measure being computed based on the performance data of advertisements assigned to the multi-dimensional index and having the combinations of attribute values for which the performance measure is being computed; and

allocating impressions for a future stage of the advertising campaign to the each advertisement in the set of advertisements based on a function of the performance measures corresponding to the combinations of attribute values for the advertisements.

108. The system of claim 107, wherein the one or more processors are further configured to perform operations including receiving data representing a number of presentations of each advertisement and a number of selections of the advertisement in response to the presentation of the advertisement.

109. The system of claim 107, wherein each of the stages of the advertisement campaign is defined by a measure of time by which performance data is delineated.

110. The system of claim 107, wherein the performance data for each previous stage is weighted based on an elapsed time since an occurrence of actions represented by the performance data.

111. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

allocating impressions to each advertisement in a set of advertisements over stages of an advertising campaign;

receiving performance data representing performance of each advertisement over the stages of the advertising campaign;

assigning each advertisement to a multi-dimensional index of advertisement attributes that is defined by a set of attributes for which the advertisement has corresponding attribute values, with each attribute in the set of attributes being an attribute that defines content of the advertisement, and each advertisement being assigned to only one multi-dimensional index;

computing performance measures for combinations of attribute values from the set of attributes, with each performance measure being computed based on the performance data of advertisements assigned to the multi-dimensional index and having the combinations of attribute values for which the performance measure is being computed; and

allocating impressions for a future stage of the advertising campaign to the each advertisement in the set of advertisements based on a function of the performance measures corresponding to the combinations of attribute values for the advertisements.

112. A method, comprising:

accessing a multi-dimensional index of advertisement attributes that is defined by a set of matching attributes for a plurality of advertisements that belong to the multi-dimensional index and performance measures for the plurality of advertisements, with the matching attributes being attributes that define content of the plurality of advertisements, and each advertisement having attribute values specified for the matching attributes;

determining a baseline performance for a first advertisement having a first set of attribute values for the set of matching attributes, the determination being based on the multi-dimensional index;

computing an estimated performance for the first advertisement when an attribute value in the first set of attribute values is adjusted, the estimated performance being computed based on the adjusted attribute value and the multi-dimensional index; and

providing a report specifying a performance change for the first advertisement resulting from the change to the attribute value, the performance change being computed based on the estimated performance relative to the baseline performance.

113. A system comprising:

a memory device storing a multi-dimensional index of advertisement attributes that is defined by a set of matching attributes for a plurality of advertisements that belong to the multi-dimensional index and performance measures for the plurality of advertisements, with the matching attributes being attributes that define content of the plurality of advertisements, and each advertisement having attribute values specified for the matching attributes; and

one or more processors configured to interact with the memory device, the one or more processors being further configured to perform operations including;

accessing the multi-dimensional index;

determining a baseline performance for a first advertisement having a first set of attribute values for the set of matching attributes, the determination being based on the multi-dimensional index;

computing an estimated performance for the first advertisement when an attribute value in the first set of attribute values is adjusted, the estimated performance being computed based on the adjusted attribute value and the multi-dimensional index; and

providing a report specifying a performance change for the first advertisement resulting from the change to the attribute value, the performance change being computed based on the estimated performance relative to the baseline performance.

114. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

accessing a multi-dimensional index of advertisement attributes that is defined by a set of matching attributes for a plurality of advertisements that belong to the multi-dimensional index and performance measures for the plurality of advertisements, with the matching attributes being attributes that define content of the plurality of advertisements, and each advertisement having attribute values specified for the matching attributes;

determining a baseline performance for a first advertisement having a first set of attribute values for the set of matching attributes, the determination being based on the multi-dimensional index;

computing an estimated performance for the first advertisement when an attribute value in the first set of attribute values is adjusted, the estimated performance being computed based on the adjusted attribute value and the multi-dimensional index; and

providing a report specifying a performance change for the first advertisement resulting from the change to the attribute value, the performance change being computed based on the estimated performance relative to the baseline performance.