Framework to evaluate content display policies
First Claim
1. A computer-implemented method comprising:
- collecting data about user activities;
wherein, the data include behavior information that indicates how each user, of a set of users, interacted with content items on a web portal during a particular period of time;
generating, based on at least some of the collected data, a plurality of user profile models, wherein each user profile model, of the plurality of user profile models, at least indicates a probability that users that match the user profile model will interact with a content item, on the web portal, having certain characteristics;
evaluating a plurality of content-selection algorithms, wherein each particular content-selection algorithm, of the plurality of content-selection algorithms, implements a different approach for selecting content items to display on the web portal, wherein each particular content-selection algorithm is not a user profile model;
wherein evaluating the plurality of content-selection algorithms comprises, for each particular content-selection algorithm of the plurality of content-selection algorithms;
using the particular content-selection algorithm to identify one or more sets of content items for display on a simulated web portal;
based on the plurality of user profile models, and at least some of the data about the user activities, simulating visits to the simulated web portal, in which simulated users select one or more content items from the identified one or more sets of content items; and
generating simulation data indicating content-item selections during the simulated visits;
wherein evaluating the plurality of content-selection algorithms further comprises, based on the simulation data from the simulated visits, computing performance metrics that measure a predicted effectiveness, of each particular content-selection algorithm of the plurality of content-selection algorithms, in choosing content items with which users will interact on the web portal;
wherein the method is performed by one or more computing devices.
9 Assignments
0 Petitions
Accused Products
Abstract
Content display policies are evaluated using two kinds of methods. In the first kind of method, using information, collected in a “controlled” manner about user characteristics and content characteristics, truth models are generated. A simulator replays users'"'"' visits to the portal web page and simulates their interactions with content items on the page based on the truth models. Various metrics are used to compare different content item-selecting algorithms. In the second kind of method, no explicit truth models are built. Events from the controlled serving scheme are replayed in part or whole; content item-selection algorithms learn using the observed user activities. Metrics that measure the overall predictive error are used to compare different content-item selection algorithms. The data collected in a controlled fashion plays a key role in both the methods.
-
Citations
31 Claims
-
1. A computer-implemented method comprising:
-
collecting data about user activities; wherein, the data include behavior information that indicates how each user, of a set of users, interacted with content items on a web portal during a particular period of time; generating, based on at least some of the collected data, a plurality of user profile models, wherein each user profile model, of the plurality of user profile models, at least indicates a probability that users that match the user profile model will interact with a content item, on the web portal, having certain characteristics; evaluating a plurality of content-selection algorithms, wherein each particular content-selection algorithm, of the plurality of content-selection algorithms, implements a different approach for selecting content items to display on the web portal, wherein each particular content-selection algorithm is not a user profile model; wherein evaluating the plurality of content-selection algorithms comprises, for each particular content-selection algorithm of the plurality of content-selection algorithms; using the particular content-selection algorithm to identify one or more sets of content items for display on a simulated web portal; based on the plurality of user profile models, and at least some of the data about the user activities, simulating visits to the simulated web portal, in which simulated users select one or more content items from the identified one or more sets of content items; and generating simulation data indicating content-item selections during the simulated visits; wherein evaluating the plurality of content-selection algorithms further comprises, based on the simulation data from the simulated visits, computing performance metrics that measure a predicted effectiveness, of each particular content-selection algorithm of the plurality of content-selection algorithms, in choosing content items with which users will interact on the web portal; wherein the method is performed by one or more computing devices. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30)
-
-
31. A system comprising:
-
one or more computing devices executing a web server that is configured to; receive requests from users; provide, in response to the requests, web pages of a web portal; and generate user logs of behavior information during user visits to the web portal, each of the user visits being associated with characteristic information about a user that is visiting the web portal; a content management system configured to select sets of content items to display on the web portal based on one or more content-selection algorithms of a plurality of content-selection algorithms; wherein each particular content-selection algorithm, of the plurality of content-selection algorithms, implements a different approach for selecting content items to display on the web portal; a user profiling component configured to generate, based on at least some of the user logs, a plurality of user profile models, wherein each user profile model of the plurality of user profile models at least indicates probabilities that users that match the user profile model will interact with certain types of content items on the web portal; a simulation component configured to simulate visits to the web portal by for each particular content-selection algorithm of the plurality of content-selection algorithms; using the particular content-selection algorithm to identify one or more sets of content items for display on a simulated web portal; based on the plurality of user profile models, and at least some of the data about the user activities, simulating visits to the simulated web portal, in which simulated users select one or more content items from the identified one or more sets of content items; and generating simulation data indicating content-item selections during the simulated visits; an algorithm evaluation component configured to, based on the simulation data from the simulated visits, compute performance metrics that measure a predicted effectiveness, of each particular content-selection algorithm of the plurality of content-selection algorithms, in choosing content items with which users will interact on the web portal.
-
Specification