Framework to evaluate content display policies

• US 8,504,558 B2
• Filed: 07/31/2008
• Issued: 08/06/2013
• Est. Priority Date: 07/31/2008
• Status: Expired due to Fees

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Agarwal, Deepak, Elango, Pradheep, Ramakrishnan, Raghu, Park, Seung-Taek, Chen, Bee-Chung
• Primary Examiner(s): Trujillo, James
• Assistant Examiner(s): VU, THONG H
• Application Number:
  US12/184,114
• Publication Number:
  US 20100030717A1
• Time in Patent Office: 1,832 Days
• Field of Search: 707/748, 707/754, 707/803, 707/1/1, 707/758, 707/721, 706/12, 706/59, 706/64
• US Class Current: 707/721
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

Content display policies are evaluated using two kinds of methods. In the first kind of method, using information, collected in a “controlled” manner about user characteristics and content characteristics, truth models are generated. A simulator replays users'"'"' visits to the portal web page and simulates their interactions with content items on the page based on the truth models. Various metrics are used to compare different content item-selecting algorithms. In the second kind of method, no explicit truth models are built. Events from the controlled serving scheme are replayed in part or whole; content item-selection algorithms learn using the observed user activities. Metrics that measure the overall predictive error are used to compare different content-item selection algorithms. The data collected in a controlled fashion plays a key role in both the methods.

Claims

1. A computer-implemented method comprising:

collecting data about user activities;

wherein, the data include behavior information that indicates how each user, of a set of users, interacted with content items on a web portal during a particular period of time;

generating, based on at least some of the collected data, a plurality of user profile models, wherein each user profile model, of the plurality of user profile models, at least indicates a probability that users that match the user profile model will interact with a content item, on the web portal, having certain characteristics;

evaluating a plurality of content-selection algorithms, wherein each particular content-selection algorithm, of the plurality of content-selection algorithms, implements a different approach for selecting content items to display on the web portal, wherein each particular content-selection algorithm is not a user profile model;

wherein evaluating the plurality of content-selection algorithms comprises, for each particular content-selection algorithm of the plurality of content-selection algorithms;

using the particular content-selection algorithm to identify one or more sets of content items for display on a simulated web portal;

based on the plurality of user profile models, and at least some of the data about the user activities, simulating visits to the simulated web portal, in which simulated users select one or more content items from the identified one or more sets of content items; and

generating simulation data indicating content-item selections during the simulated visits;

wherein evaluating the plurality of content-selection algorithms further comprises, based on the simulation data from the simulated visits, computing performance metrics that measure a predicted effectiveness, of each particular content-selection algorithm of the plurality of content-selection algorithms, in choosing content items with which users will interact on the web portal;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein the web portal is generated by a content publishing system which selects one or more content items for display on the web portal in accordance with at least a particular content-selection algorithm of the plurality of content-selection algorithms.

3. The method of claim 1, further comprising simulating how a particular user interacts with a particular set of one or more content items selected using a particular content-selection algorithm of the plurality of content-selection algorithms, wherein the simulating is performed without ever actually presenting the particular set of one or more content items to the particular user.

4. The method of claim 1, wherein the step of collecting data comprises storing at least one of:

(a) a gender of a user, (b) an age of a user, (c) a geographical location of a user, (d) one or more interests of a user, (e) information about previous user activities in relation to a web page of the content publishing system, (f) information about a user'"'"'s previous activities in relation to several web pages of a web portal, (g) information about a user'"'"'s activities off other portals and networks, or (h) information about a user'"'"'s social network.

5. The method of claim 1, wherein the step of collecting data comprises storing at least one of:

(a) information that indicates a time of day that a user accessed a web page, (b) information that indicates a duration of time that a user viewed the web page, or (c) information that identifies links that are both on the web page and on which a user clicked.

6. The method of claim 1, a user profile model of the plurality of user profile models includes one or more of:

(a) user characteristics, (b) content characteristics of content items requested by the user, including one or more of;

content item identifiers, title keywords and/or phrases, content keywords and/or phrases, named entities, or content categories, (c) context information for activities taken by the user, including one or more of a position of a content item selected by the user on the web portal at the time of the content item being selected, the time when the user selected the content item, or other articles displayed on the web portal at the time.

7. The method of claim 1, wherein the probability is based on a ratio of (a) how many of the users also clicked, during a particular period of time, on a content item having the certain characteristics, to (b) how many of the users accessed the web portal during the particular period of time.

8. The method of claim 1, further comprising evaluating actual effectiveness of one or more content item-selection algorithms of the plurality of content-selection algorithms based on future data collected about user activities, wherein the actual effectiveness of the one or more content item-selection algorithms is measured cumulatively over several time intervals.

9. A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform the steps recited in claim 8.

10. The method of claim 1, wherein evaluating the plurality of content-selection algorithms further comprises:

for each particular user of at least a subset of the set of users, determining which particular user profile model, of the plurality of user profile models, that the particular user'"'"'s characteristic information matches, and simulating, based at least in part on the probability specified by the particular user profile model, how the particular user interacts with different sets of content items selected using different content-selection algorithms in the simulated visits to the simulated web portal.

11. The method of claim 1,

wherein the data collected about the user activities comprises a log of previous visits to the web portal during a particular period of time;

wherein simulating the visits comprises, for each previous visit of the previous visits, matching the previous visit to a particular user profile model and simulating, based the user profile model, how a user matching the particular profile model would have interacted with different sets of content items selected different content-selection algorithms, of the plurality of content-selection algorithms.

12. The method of claim 1, wherein identifying the one or more sets of content items comprises associating each set of content items of the one or more sets of content items with ordering information indicating at which position each content item in the sets of content items would have been displayed in the simulated web portal, wherein simulating the visits is also based on the ordering information.

13. The method of claim 1, wherein simulating the visits comprises:

tracking a particular content item that was selected during a first simulated visit for a particular simulated user;

preventing the particular content item from being selected during a second simulated visit for the particular simulated user.

14. The method of claim 1, further comprising evaluating the plurality of content-selection algorithms by simulating visits repeatedly over time, wherein the performance metrics are based on repeatedly simulating the visits.

15. The method of claim 1, wherein simulating the visits comprises constraining the visits by certain business rules, whereby the predicted effectiveness of each particular content-selection algorithm of the plurality of content-selection algorithms is evaluated relative to the certain business rules.

16. The method of claim 1, wherein simulating a particular visit of the simulated visits comprises determining whether a particular simulated user, matching a particular user profile model, selects a particular content item at a particular position within the simulated web portal based on at least on a randomly selected number and a particular probability associated with the particular content item and the particular user profile model.

17. A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform the steps recited in claim 1.

18. A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform the steps recited in claim 2.

19. A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform the steps recited in claim 3.

20. A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform the steps recited in claim 4.

21. A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform the steps recited in claim 5.

22. A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform the steps recited in claim 6.

23. A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform the steps recited in claim 7.

24. A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform the steps recited in claim 10.

25. A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform the steps recited in claim 11.

26. A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform the steps recited in claim 12.

27. A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform the steps recited in claim 13.

28. A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform the steps recited in claim 14.

29. A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform the steps recited in claim 15.

30. A non-transitory computer-readable storage medium storing one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to perform the steps recited in claim 16.

31. A system comprising:

one or more computing devices executing a web server that is configured to;

receive requests from users;

provide, in response to the requests, web pages of a web portal; and

generate user logs of behavior information during user visits to the web portal, each of the user visits being associated with characteristic information about a user that is visiting the web portal;

a content management system configured to select sets of content items to display on the web portal based on one or more content-selection algorithms of a plurality of content-selection algorithms;

wherein each particular content-selection algorithm, of the plurality of content-selection algorithms, implements a different approach for selecting content items to display on the web portal;

a user profiling component configured to generate, based on at least some of the user logs, a plurality of user profile models, wherein each user profile model of the plurality of user profile models at least indicates probabilities that users that match the user profile model will interact with certain types of content items on the web portal;

a simulation component configured to simulate visits to the web portal by for each particular content-selection algorithm of the plurality of content-selection algorithms;

using the particular content-selection algorithm to identify one or more sets of content items for display on a simulated web portal;

based on the plurality of user profile models, and at least some of the data about the user activities, simulating visits to the simulated web portal, in which simulated users select one or more content items from the identified one or more sets of content items; and

generating simulation data indicating content-item selections during the simulated visits;

an algorithm evaluation component configured to, based on the simulation data from the simulated visits, compute performance metrics that measure a predicted effectiveness, of each particular content-selection algorithm of the plurality of content-selection algorithms, in choosing content items with which users will interact on the web portal.