System and method for improving the performance of electronic media advertising campaigns through multi-attribute analysis and optimization

• US 7,406,434 B1
• Filed: 12/17/2001
• Issued: 07/29/2008
• Est. Priority Date: 12/15/2000
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.), IQ Acquisition Corporation
• Original Assignee: Carl Meyer
• Inventors: Weisinger, James R., Kumar, Sachin, Ranka, Sanjay, Chang, E. Diane, Chatwin, Richard E., Lenderman, Jason
• Primary Examiner(s): BOYCE, ANDRE D
• Application Number:
  US10/024,449
• Time in Patent Office: 2,416 Days
• Field of Search: 705/10
• US Class Current: 705/14.43
• CPC Class Codes: G06Q 10/06375 Prediction of business process outcome or impact based on a proposed change
• CPC Class Codes: G06Q 10/06393 Score-carding, benchmarking or key performance indicator [KPI] analysis
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0203 Market surveys; Market polls
• CPC Class Codes: G06Q 30/0204 Market segmentation
• CPC Class Codes: G06Q 30/0207 Discounts or incentives, e.g. coupons or rebates
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06Q 30/0242 Determining effectiveness of advertisements
• CPC Class Codes: G06Q 30/0243 Comparative campaigns
• CPC Class Codes: G06Q 30/0244 Optimization
• CPC Class Codes: G06Q 30/0277 Online advertisement

Abstract

Automated system, methods, algorithms, procedures, and computer software programs and computer program products for improving and optimizing the performance of messaging campaigns, particularly for marketing campaigns in which advertisements or other messages are distributed over an interactive measurable medium such as the Internet. Analysis and Optimization method and procedure, an automated system, and system and method that exploit the underlying multi-attribute structure, as well as other features and advantages. Optimization procedures allocate the ad alternatives or other message to the customer population to optimize business objectives such as maximizing the number of positive responses received. Procedure for generating message allocations that improve and attempt to optimize the campaign performance. Methods ensure that campaign constraints are not violated. Methods can be implemented on a computer that is programmed to retrieve message performance information and to generate recommended message allocations for each stage in a multi-stage messaging campaign to achieve messaging goals.

Claims

1. A method of multiattribute analysis and optimization for providing automated measurements of an importance of attributes and attribute values of message alternatives and, in an automated manner, for improving a stage-to-stage performance of the message alternatives in a multi-stage message campaign in an interactive measurable medium;

the method comprising;

obtaining historical information including historical performance data for message performance for at least one previous stage of the multi-stage message campaign;

generating a multiattribute system that describes the message alternatives in terms of the attributes and the attribute values, wherein an attribute is an inherent characteristic of a message alternative and an attribute value is a particular instantiation of the attribute, wherein the multiattribute system is generated by grouping the message alternatives into one or more multiattribute structures, wherein each multiattribute structure is a collection of attributes and each message alternative is assigned to one and only one multiattribute structure or a null multiattribute structure, and wherein, for each such message alternative, there is one and only one attribute value for each attribute that comprises the multiattribute structure to which the message alternative is assigned;

determining an importance of the attributes and the attribute values to the performance of the message alternatives;

based on the determination of the importance of the attributes and the attribute values, optimizing campaign performance through allocation of message alternatives to visitors during the multi-stage message campaign; and

determining a next allocation for each new visitor for a next stage of the campaign.

2. A method as in claim 1, further comprising:

processing said historical performance data to identify erroneous or possibly erroneous or unusual historical performance data.

3. A method as in claim 2, wherein said processing further including cleaning said historical performance data to correct said identified erroneous or possible erroneous or unusual historical performance data.

4. A method as in claim 3, further comprising discounting at least a portion of said historical performance data from said at least one previous stage to grant more weight to more recent historical performance data than to less recent historical performance data.

5. A method as in claim 1, further comprising discounting at least a portion of said historical performance data from said at least one previous stage to grant more weight to more recent historical performance data than to less recent historical performance data.

6. A method as in claim 5, wherein said historical information further includes historical information selected from the set consisting of a visitor identifier, a message alternative identifier for the message alternative shown to said visitor, a type of action or types of actions taken by each said visitor, and a payoff for each of the actions taken by said visitors.

7. A method as in claim 1, wherein said historical information further includes historical information selected from the set consisting of a visitor identifier, a message alternative identifier for the message alternative shown to said visitor, a type of action or actions taken by each said visitor, and a payoff for each of the action or actions taken by said visitors.

8. A method as in claim 6, wherein said historical information further includes additional information selected from the set consisting of:

a message type, a message type and a corresponding alternative clicked on by a visitor, a web site and a section of the web site where the visitor was presented with and saw or had an opportunity to see a banner ad, a time of the visitors visit, a visitor demographic information, a visitor psychographic information, a visitor demographic profile, a visitor psychographic profile, and combinations thereof.

9. A method as in claim 7, wherein said historical information further includes additional information selected from the set consisting of:

a message type, a message type and a corresponding alternative clicked on by a visitor, a web site and a section of the web site where the visitor was presented with and saw or had an opportunity to see a banner ad, a time of the visitors visit, a visitor demographic information, a visitor psychographic information, a visitor demographic profile, a visitor psychographic profile, and combinations thereof.

10. A method as in claim 1, wherein said historical information includes additional information selected from the set consisting of:

a message type, a message type and a corresponding alternative clicked on by a visitor, a web site and a section of the web site where the visitor was presented with and saw or had an opportunity to see a banner ad, a time of the visitors visit, a visitor demographic information, a visitor psychographic information, a visitor demographic profile, a visitor psychographic profile, and combinations thereof.

11. A method as in claim 1, further comprising:

processing said historical data to identify and correct any erroneous data;

discounting at least a portion of said performance data from said at least one previous stage to grant more weight to more recent performance data than to less recent performance data; and

said historical information further includes;

information selected from the set consisting of;

a visitor identifier, a message alternative identifier for the message alternative shown to said visitor, a type of action or actions taken by said visitor, and a payoff for each of the actions taken by said visitors, a message type, a message type and a corresponding alternative clicked on by a visitor, a web site and a section where the visitor was presented and saw a banner ad, a time of the visitors visit, a visitor demographic information, a visitor psychographic information, a visitor demographic profile, a visitor psychographic profile, and combinations thereof.

12. A method as in claim 11, wherein said message alternative is selected from a group of message alternatives consisting of:

an email, an ad, a banner ad, a banner, a splash page, a home page, a jump page, a landing page, media programming, media content, a political campaign message, a survey, a poll, a news headline, a headline, a ballot measure, a ballot initiative, one or more sports scores, one or more sports scores for a local, regional, collegiate, or amateur sports team or teams, and combinations thereof.

13. A method as in claim 1, wherein said message alternative is selected from a group of message alternatives consisting of:

an email, an ad, a banner ad, a splash page, a home page, a jump page, a landing page, media programming, media content, a political campaign message, a survey, a poll, a news headline, a headline, a ballot measure, a ballot initiative, one or more sports scores, one or more sports scores for a local, regional, collegiate, or amateur sports team or teams, and combinations thereof.

14. A method as in claim 1, wherein each exposure of a message alternative to a visitor results in a payoff result, a performance of said message alternative for a visitor segment being based at least in part on the expected value of the payoff results for the visitors for that visitor segment, said payoff result being a function of a measurement including at least one measurement selected from the set of measurements consisting of:

a number of actions, a number of signups, a number of purchases, a binary result, a value of purchases, a revenue amount, a sales amount, a profit amount, a continuous function of a continuous variable, a continuous function of a discontinuous variable, a continuous function of a binary variable, a discrete function of a continuous variable, a discrete function of a discontinuous variable, a discrete function of a binary variable, and combinations thereof.

15. A method as in claim 12, wherein each exposure of a message alternative to a visitor results in a payoff result, a performance of said message alternative for a visitor segment being based at least in part on the expected value of the payoff results for the visitors for that visitor segment, said payoff result being a function of a measurement including at least one measurement selected from the set of measurements consisting of:

a number of actions, a number of signups, a number of purchases, a binary result, a value of purchases, a revenue amount, a sales amount, a profit amount, a continuous function of a continuous variable, a continuous function of a discontinuous variable, a continuous function of a binary variable, a discrete function of a continuous variable, a discrete function of a discontinuous variable, a discrete function of a binary variable, and combinations thereof.

16. A method as in claim 14, wherein said multiattribute method comprises:

generating at least one report on an optimal value for each attribute;

generating new message alternatives with the values of each attribute indicated by a multiattribute analysis that uses prior expectations and payoff criteria to arrive at message alternative evaluation;

adding said new message alternatives to the set of available messages to be optimized; and

choosing the best message alternative from among the choices presented and proactively creating message alternatives that are expected to perform well within some confidence level.

17. A method as in claim 16, wherein said multiattribute system is of standard form.

18. A method as in claim 16, wherein said multiattribute system is reduced to standard form.

19. A method as in claim 16, wherein said processing includes discounting said performance data;

and said discounting being achieved using at least one of discounting scheme selected from the set of discounting schemes consisting of;

a geometric discounting scheme, a linear weighting discounting scheme, a non-linear weighting discounting scheme, and combinations thereof.

20. A method as in claim 1, wherein said processing includes discounting said performance data;

and said discounting being achieved using at least one of discounting scheme selected from the set consisting of;

a geometric discounting scheme, a linear weighting discounting scheme, a non-linear weighting discounting scheme, and combinations thereof.

21. A method as in claim 19, wherein said processing includes estimating the distribution of the expected payoff of future performance of said message alternatives.

22. A method as in claim 21, wherein said estimate is obtained under the assumption that the underlying performance model implied by said multiattribute system holds.

23. A method as in claim 21, wherein said estimate is obtained by first estimating the distribution of expected payoff of future performance of said message alternatives assuming said underlying performance model implied by said multiattribute system holds, then updating the second said estimate to remove said assumption.

24. A method as in claim 21, wherein said estimate is employed for the purpose of allocating future campaign impressions amongst said message alternatives to optimize future expected campaign performance.

25. A method as in claim 19, wherein said allocation method comprises performing a pairwise comparison procedure among said plurality of message alternatives.

26. A method as in claim 25, wherein said pairwise comparison comprises:

computing an expected value of a payoff distribution of each message alternative and an error in said payoff distribution expected value; and

comparing in pairwise manner the expected value based statistic of a selected message alternative with the same expected value based statistic for each other of the message alternatives.

27. A method as in claim 26, wherein said pairwise comparison comprises:

selecting a first message alternative and a second message alternative from among said plurality of message alternatives;

computing a quotient determined as difference between the estimate of the said expected value divided by a standard error in the difference between the estimate of the expected value for each of said first and second message alternatives; and

determining if said difference is within a predetermined magnitude relationship of, including greater than, a threshold cutoff value c₁, and;

(i) if said difference is within said predetermined magnitude relationship of said threshold cutoff value, then identifying the second selected message alternative as a member of a non-contending message class.

28. A method as in claim 27, wherein each message alternative is given content, the method further comprising:

repeating said selecting, said computing a quotient, said computing a difference, and said determining for other all other pairs of said message alternatives on a pairwise basis so that when all possible pairwise comparisons have been made each message alternative will have been sorted into one contender and said non-contender classes.

29. A method as in claim 28, further comprising performing said repeating for all other pairs of message alternatives on a pairwise basis for each optimization.

30. A method as in claim 28, wherein said pairwise comparison procedure further comprises initially uniformly allocating said message alternatives prior to performing a first optimization.

31. A method as in claim 27, wherein the message alternatives are sorted into a contender class and a non-contender class, and for at least one identified stage in a messaging campaign all visitors are allocated equally among message alternatives belonging to the contender class, and no visitors are allocated to message alternatives belonging to the non-contender class.

32. A method as in claim 31, wherein said at least one identified stage comprises an initial stage.

33. A method as in claim 31, wherein said at least one identified stage comprises a plurality of stages.

34. A method as in claim 31, wherein said at least one identified stage comprises all stages.

35. A method as in claim 25, wherein said allocation comprises a pairwise comparison, and message alternatives are sorted into more than two classes.

36. A method as in claim 35, wherein said messages belonging to each said class are assigned different numbers of visitors.

37. A method as in claim 35, wherein said messages belonging to each said class are assigned substantially the same numbers of visitors.

38. A method as in claim 25, wherein said allocation comprises a pairwise comparison, and message alternatives are sorted into more than two classes and the allocation among contenders is not equal.

39. A method as in claim 25, wherein said allocation comprises a pairwise comparison, and message alternatives are sorted into more than two classes and the allocation among non-contenders is not equal.

40. A method as in claim 25, wherein said allocation comprises a pairwise comparison, and message alternatives are sorted into more than two classes and the allocation among contenders is not equal and the allocation among non-contenders is not equal.

41. A method as in claim 27, wherein said allocation comprises a pairwise comparison, and the cutoff c₁ is in the range substantially between 0-1.0.

42. A method as in claim 27, wherein said allocation comprises a pairwise comparison, and the cutoff c₁ is substantially 0.5.

43. A method as in claim 27, wherein said allocation comprises a pairwise comparison, and the cutoff c₁ is greater than or equal to 1.

44. A method as in claim 27, wherein said allocation comprises a pairwise comparison, and the cutoff c₁ is greater than 1.

45. A method as in claim 16, wherein reports on the optimal value for each attribute are provided only if it is determined that said multiattribute system supports said measurement of the relative importance to said message alternative performance of said attributes and said attribute values.

46. A method as in claim 45, wherein said multiattribute system is reduced to standard form to determine if said multiattribute system supports said measurement of the relative importance to said message alternative performance of said attributes and said attribute values.

47. A method as in claim 45, wherein determination of said relative importance to said message alternative performance of said attributes and said attribute values includes determination of confidence intervals around said relative importance.

48. A method as in claim 16, wherein reports on the optimal value for each attribute are provided upon a given occurrence.

49. A method as in claim 48, wherein an acceptance/rejection test is employed upon a given occurrence.

50. A method as in claim 48, wherein reports on said relative importance to said message alternative performance of said attributes and said attribute values are provided to the marketing manager or other interested party.

51. A method as in claim 48, wherein determination of said relative importance to said message alternative performance of said attributes and said attribute values includes determination of confidence intervals around said relative importance.

52. A method as in claim 50, wherein said performance measurement represents click-through rates of the message alternatives, the multiattribute function is a logodds function, and the reported relative importance to the message alternative performance of one of the attribute values is computed as 100*(exp{u}−

1), where u represents a parameter estimate of a multiattribute parameter associated with the attribute value, and the reported relative importance to the message alternative performance of one of the attributes is computed as 100×

(exp{u}−

1), where u represents the largest parameter estimate of a multiattribute parameter associated with the attribute values of the attribute.

53. A method as in claim 52, wherein said reports include a chart and a table.

54. A method as in claim 53, wherein said chart is a bar chart where the length of the bars are determined by said reported relative importance of said attributes or said attribute values.

55. A method as in claim 54, wherein the colors of said bars are determined by assigning each attribute a color, and whenever an attribute or a value of said attribute appears on a chart, coloring said bar with said color.

56. A method as in claim 54, wherein the colors of said bars are determined by, when reporting on an attribute, using one color if the lower confidence bound of the best attribute value for said attribute is less than zero, and using a second color otherwise;

and likewise when reporting on an attribute value, using one color if the lower confidence bound of said attribute value is less than zero, and using a second color otherwise.

57. A method as in claim 54, wherein the colors of said bars are determined by assigning each attribute a color and each attribute value a variant of the color assigned to its attribute, and whenever an attribute or an attribute value appears on a chart, coloring said bar with said color.

58. The method as described in claim 1 further including reducing the multiattribute system to a standard form such that (i) each multiattribute structure, except any null multiattribute structure, includes two or more attributes, (ii) each attribute includes at least two or more attribute values, (iii) for every attribute in each multiattribute structure at least two attribute values are assigned to message alternatives in that multiattribute structure, and (iv) a multiattribute mapping matrix for each multiattribute structure has full column rank.

59. A computer program product for use in conjunction with a computer system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism comprising:

a program module that directs the computer system and/or components thereof, to function in a specified manner to provide automated measurements of an importance of attributes and attribute values of message alternatives and, in an automated manner, to improve a stage-to-stage performance of the message alternatives in a multi-stage message campaign in an interactive measurable medium, the program module including instructions for;

obtaining historical information including performance data for message performance for at least one previous stage of the multi-stage message campaign;

generating a multiattribute system that describes the message alternatives in terms of the attributes and the attribute values, wherein an attribute is an inherent characteristic of a message alternative and an attribute value is a particular instantiation of the attribute, wherein the multiattribute system is generated by grouping the message alternatives into one or more multiattribute structures, wherein each multiattribute structure is a collection of attributes and each message alternative is assigned to one and only one multiattribute structure or a null multiattribute structure, and wherein, for each such message alternative, there is one and only one attribute value for each attribute that comprises the multiattribute structure to which the message alternative is assigned;

determining an importance of the attributes and the attribute values to the performance of the message alternatives;

based on the determination of the importance of the attributes and the attribute values, optimizing campaign performance through allocation of message alternatives to visitors during the multi-stage message campaign; and

determining a next allocation for each new visitor for a next stage of the campaign.

60. A computer program product as in claim 59, wherein said program module further including instructions for:

processing said historical data to identify and correct any erroneous data;

discounting at least a portion of said performance data from said at least one previous stage to grant more weight to more recent performance data than to less recent performance data; and

wherein;

said historical information further includes additional information selected from the set consisting of;

a visitor identifier, a message alternative identifier for the message alternative shown to said visitor, a type of action or actions taken by said visitor, and a payoff for each of the actions taken by said visitors, a message type, a message type and a corresponding alternative clicked on by a visitor, a web site and a section where the visitor was presented and saw a banner ad, a time of the visitors visit, a visitor demographic information, a visitor psychographic information, a visitor demographic profile, a visitor psychographic profile, and combinations thereof.

61. A computer program product as in claim 59, wherein said message alternatives are selected from a group consisting of:

an email, an ad, a banner ad, a splash page, a home page, a jump page, a landing page, media programming, media content, a political campaign message, a survey, a poll, a news headline, a headline, a ballot measure, a ballot initiative, one or more sports scores, one or more sports scores for a local, regional, collegiate, or amateur sports team or teams, and combinations thereof.

62. The computer program product as described in claim 59 wherein the program module includes instructions for reducing the multiattribute system to a standard form such that (i) each multiattribute structure, except any null multiattribute structure, includes two or more attributes, (ii) each attribute includes at least two or more attribute values, (iii) for every attribute in each multiattribute structure at least two attribute values are assigned to message alternatives in that multiattribute structure, and (iv) a multiattribute mapping matrix for each multiattribute structure has full column rank.