Determining better ad selection, scoring, and/or presentation techniques

• US 9,858,590 B1
• Filed: 12/30/2003
• Issued: 01/02/2018
• Est. Priority Date: 12/30/2003
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Dean, Jeffrey A., Harik, Georges R., Shazeer, Noam, Tong, Simon
• Primary Examiner(s): Stoltenberg, David
• Assistant Examiner(s): WOODWORTH, II, ALLAN J
• Application Number:
  US10/748,744
• Time in Patent Office: 5,117 Days
• Field of Search: None
• CPC Class Codes: G06Q 30/0244 Optimization

Abstract

Different ad selection techniques may be evaluated and compared by (i) combining ads generated using at least two different techniques and (ii) determining performance statistics of these combined (e.g., interleaved) advertising serves over time. The relative performance of the different techniques can then be determined. These principles can also be used to gauge different ad scoring techniques. These principles can also be used to gauge different ad rendering techniques.

Claims

1. A method for determining performance of first and second ad generation techniques in an ad serving computer network, the method comprising:

obtaining a first set of one or more ads, the first set being generated using a first ad generation technique and ad request information, the first ad generation technique including a first ad selection technique, a first ad scoring technique, or a first ad rendering technique, each ad in the first set having a first rank score associated therewith;

obtaining a second set of one or more ads, the second set being generated using a second ad generation technique and the ad request information, the second ad generation technique including a second ad selection technique, a second ad scoring technique, or a second ad rendering technique, each ad in the second set having a second rank score associated therewith;

preparing, by a processor, an interleaved order of ads from the first and second sets by;

selecting one of the first ad generation technique and the second ad generation technique as an initial technique for the interleaved order;

adding to a list of ads, from the selected initial technique, a next highest scoring ad not already included in the list of ads;

adding to the list of ads, from the ad generation technique not selected by the processor, another next highest scoring ad not already included in the list of ads; and

automatically repeating the adding operations in accordance with the interleaved order until the list of ads meets a threshold condition;

in accordance with one or more serving parameters that specify (i) a webpage on which an ad is served, (ii) a search query or a search result associated with serving of the ad, (iii) a user characteristic specifying a type of browser used, one or more previous webpage views, or previous online behavior, or (iv) a host or affiliate site that initiated an ad request, serving a plurality of ads from the list of ads; and

determining the performance of the first and second ad generation techniques by monitoring, over one or more networks, an online performance of each of the plurality of ads served, including updating a data repository with data specifying the online performance of each of the plurality of ads served and a respective indicator of that ad, the respective indicator identifying which ad generation technique produced that ad, wherein, when a particular ad of the plurality of ads served is in both the first and second sets of ads, the online performance of the particular ad is either associated with both the first and second ad generation techniques or is not associated with either of the first and second ad generation techniques.

2. The method of claim 1, wherein the selection of the initial technique for the interleaved order is a random selection.

3. The method of claim 1, wherein the interleaved order alternates back and forth between a single ad from the selected ad generation technique followed by a single ad from the not selected ad generation technique.

4. The method of claim 1, wherein the interleaved order alternates back and forth between two or more consecutive ads from the selected ad generation technique followed by two or more consecutive ads from the not selected ad generation technique.

5. The method of claim 1, wherein serving the plurality of ads includes interleaving a spatial position of the ads from the first set of ads and the second set of ads on a document to be displayed.

6. The method of claim 5 wherein the document is a search results page.

7. The method of claim 5 wherein the document is a Web page.

8. The method of claim 5 wherein the document is an e-mail.

9. The method of claim 1, wherein serving the plurality of ads includes interleaving a temporal position of the ads from the first set of ads and the second set of ads.

10. The method of claim 1, wherein the ads from the first set of ads are ordered using the first rank scores of the first set of ads, and the ads from the second set of ads are ordered using the second rank scores of the second set of ads.

11. The method of claim 1, wherein the first ad generation technique uses a different ad selection technique than the second ad generation technique.

12. The method of claim 11 wherein the first ad generation technique uses a same ad scoring technique as the second ad generation technique.

13. The method of claim 12 wherein the first ad generation technique uses a same ad rendering technique as the second ad generation technique.

14. The method of claim 12 wherein the first ad generation technique uses a different ad rendering technique than the second ad generation technique.

15. The method of claim 11 wherein the first ad generation technique uses a different ad scoring technique than the second ad generation technique.

16. The method of claim 15 wherein the first ad generation technique uses a same ad rendering technique as the second ad generation technique.

17. The method of claim 15 wherein the first ad generation technique uses a different ad rendering technique than the second ad generation technique.

18. The method of claim 11 wherein the first ad generation technique uses a different ad rendering technique than the second ad generation technique.

19. The method of claim 11 wherein the first ad generation technique uses a same ad rendering technique as the second ad generation technique.

20. The method of claim 19 wherein the first ad generation technique uses a same ad scoring technique as the second ad generation technique.

21. The method of claim 1, wherein the first ad generation technique uses a same ad scoring technique as the second ad generation technique, but a different ad rendering technique.

22. The method of claim 21 wherein the first ad generation technique uses a same ad selection technique as the second ad generation technique.

23. The method of claim 1, wherein the first ad generation technique uses a same ad rendering technique as the second ad generation technique, but a different ad scoring technique.

24. The method of claim 23 wherein the first ad generation technique uses a same ad selection technique as the second ad generation technique.

25. The method of claim 1 wherein the first ad generation technique uses a different ad scoring technique and a different ad rendering technique than the second ad generation technique.

26. The method of claim 25 wherein the first ad generation technique uses a same ad selection technique as the second ad generation technique.

27. The method of claim 1, wherein the first ad generation technique uses a different ad scoring technique than the second ad generation technique, and wherein the performance of the first ad generation technique and the second ad generation technique is monitored by monitoring click-through performance of the served plurality of ads.

28. The method of claim 1, wherein the first ad generation technique uses a different ad scoring technique than the second ad generation technique, and wherein the performance of the first ad generation technique and the second ad generation technique is monitored by monitoring conversion performance of the served plurality of ads.

29. The method of claim 1, wherein the first ad generation technique uses a different ad selection technique than the second ad generation technique, and wherein the performance of the first ad generation technique and the second ad generation technique is monitored by monitoring click-through performance of the served plurality of ads.

30. The method of claim 1, wherein the first ad generation technique uses a different ad selection technique than the second ad generation technique, and wherein the performance of the first ad generation technique and the second ad generation technique is monitored by monitoring conversion performance of the served plurality of ads.

31. The method of claim 1, wherein the first ad generation technique uses a different ad rendering technique than the second ad generation technique, and wherein the performance of the first ad generation technique and the second ad generation technique is monitored by monitoring click-through performance of the served plurality of ads.

32. The method of claim 1, wherein the first ad generation technique uses a different ad rendering technique than the second ad generation technique, and wherein the performance of the first ad generation technique and the second ad generation technique is monitored by monitoring conversion performance of the served plurality of ads.

33. An apparatus for determining performance of first and second ad generation techniques in an ad serving computer network, the apparatus comprising:

a memory configured to store a list of ads;

a processor coupled to the memory, the processor being configured to;

obtain a first set of one or more ads, the first set having been generated using a first ad generation technique and ad request information, the first ad generation technique including a first ad selection technique, a first ad scoring technique, or a first ad rendering technique, each ad in the first set having a first rank score associated therewith;

obtain a second set of one or more ads, the second set having been generated using a second ad generation technique and the ad request information, the second ad generation technique including a second ad selection technique, a second ad scoring technique, or a second ad rendering technique, each ad in the second set having a second rank score associated therewith;

prepare an interleaved order of ads from the first and second sets by;

selecting one of the first ad generation technique and the second ad generation technique as an initial technique for the interleaved order;

adding to the list of ads, from the selected initial technique, a next highest scoring ad not already included in the list of ads;

adding to the list of ads, from the ad generation technique not selected by the processor, another next highest scoring ad not already included in the list of ads; and

automatically repeating the adding operations in accordance with the interleaved order until the list of ads meets a threshold condition;

in accordance with one or more serving parameters that specify (i) a webpage on which an ad is served, (ii) a search query or a search result associated with serving of the ad, (iii) a user characteristic specifying a type of browser used, one or more previous webpage views, or previous online behavior, or (iv) a host or affiliate site that initiated an ad request, serve a plurality of ads from the list of ads; and

determine the performance of the first and second ad generation techniques by monitoring, over one or more networks, an online performance of the plurality of ads served, including updating a data repository with data specifying the online performance of each of the plurality of ads served and a respective indicator of that ad, the respective indicator identifying which ad generation technique produced that ad, wherein, when a particular ad of the plurality of ads served is in both the first and second sets of ads, the online performance of the particular ad is either associated with both the first and second ad generation techniques or is not associated with either of the first and second ad generation techniques.

34. The apparatus of claim 33, wherein the selection of the initial technique for the interleaved order is a random selection.

35. The apparatus of claim 33, wherein the interleaved order alternates back and forth between a single ad from the selected ad generation technique followed by a single ad from the not selected ad generation technique.

36. The apparatus of claim 33, wherein the interleaved order alternates back and forth between two or more consecutive ads from the selected ad generation technique followed by two or more consecutive ads from the not selected ad generation technique.

37. The apparatus of claim 33, wherein the processor serves the plurality of ads by interleaving a spatial position of the ads from the first set of ads and the second set of ads on a document to be displayed.

38. The apparatus of claim 37 wherein the document is a search results page.

39. The apparatus of claim 37 wherein the document is a Web page.

40. The apparatus of claim 37 wherein the document is an e-mail.

41. The apparatus of claim 33, wherein the processor serves the plurality of ads by interleaving a temporal position of the ads from the first set of ads and the second set of ads.

42. The apparatus of claim 33, wherein the ads from the first set of ads are ordered using the first rank scores of the first set of ads, wherein the ads from the second set of ads are ordered using the second rank scores of the second set of ads.

43. The apparatus of claim 33, wherein the first ad generation technique uses a different ad selection technique than the second ad generation technique.

44. The apparatus of claim 43 wherein the first ad generation technique uses a same ad scoring technique as the second ad generation technique.

45. The apparatus of claim 44 wherein the first ad generation technique uses a same ad rendering technique as the second ad generation technique.

46. The apparatus of claim 44 wherein the first ad generation technique uses a different ad rendering technique than the second ad generation technique.

47. The apparatus of claim 43 wherein the first ad generation technique uses a different ad scoring technique than the second ad generation technique.

48. The apparatus of claim 47 wherein the first ad generation technique uses a same ad rendering technique as the second ad generation technique.

49. The apparatus of claim 47 wherein the first ad generation technique uses a different ad rendering technique than the second ad generation technique.

50. The apparatus of claim 43 wherein the first ad generation technique uses a different ad rendering technique than the second ad generation technique.

51. The apparatus of claim 43 wherein the first ad generation technique uses a same ad rendering technique as the second ad generation technique.

52. The apparatus of claim 51 wherein the first ad generation technique uses a same ad scoring technique as the second ad generation technique.

53. The apparatus of claim 33, wherein the first ad generation technique uses a same ad scoring technique as the second ad generation technique, but a different ad rendering technique.

54. The apparatus of claim 53 wherein the first ad generation technique uses a same ad selection technique as the second ad generation technique.

55. The apparatus of claim 33, wherein the first ad generation technique uses a same ad rendering technique as the second ad generation technique, but a different ad scoring technique.

56. The apparatus of claim 55 wherein the first ad generation technique uses a same ad selection technique as the second ad generation technique.

57. The apparatus of claim 33, wherein the first ad generation technique uses a different ad scoring technique and a different ad rendering technique than the second ad generation technique.

58. The apparatus of claim 57 wherein the first ad generation technique uses a same ad selection technique as the second ad generation technique.

59. The apparatus of claim 33, wherein the first ad generation technique uses a different ad scoring technique than the second ad generation technique, and wherein the processor is further configured to monitor click-through performance of the served plurality of ads.

60. The apparatus of claim 33, wherein the first ad generation technique uses a different ad scoring technique than the second ad generation technique, and wherein the processor is further configured to monitor conversion performance of the served plurality of ads.

61. The apparatus of claim 33, wherein the first ad generation technique uses a different ad selection technique than the second ad generation technique, and wherein the processor is further configured to monitor click-through performance of the served plurality of ads.

62. The apparatus of claim 33, wherein the first ad generation technique uses a different ad selection technique than the second ad generation technique, and wherein the processor is further configured to monitor conversion performance of the served plurality of ads.

63. The apparatus of claim 33, wherein the first ad generation technique uses a different ad rendering technique than the second ad generation technique, and wherein the processor is further configured to monitor click-through performance of the served plurality of ads.

64. The apparatus of claim 33, wherein the first ad generation technique uses a different ad rendering technique than the second ad generation technique, and wherein the processor is further configured to monitor conversion performance of the served plurality of ads.