System and method for indexing web content using click-through features

• US 7,647,314 B2
• Filed: 04/28/2006
• Issued: 01/12/2010
• Est. Priority Date: 04/28/2006
• Status: Active Grant

• Current Assignee: Pinterest, Inc.
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Zheng, Zhaohui, Sun, Gordon
• Primary Examiner(s): Jalil; Neveen Abel
• Assistant Examiner(s): LIANG, VEI CHUNG
• Application Number:
  US11/414,405
• Publication Number:
  US 20070255689A1
• Time in Patent Office: 1,355 Days
• Field of Search: None
• US Class Current: 707/742
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

System and method for the determination of the relevance of a content item to a query through the use of a machine learned relevance function that incorporates click-through features of the content items. A method for selecting a relevance function to determine a relevance of a query-content item pair comprises generating training set having one or more query-URL pairs labeled for relevance based on their click-through features. The labeled query-URL pairs are used to determine the relevance function by minimizing a loss function that accounts for click-through features of the content item. The computed relevance function is then applied to the click-through features of unlabeled content items to assign relevance scores thereto. An inverted click-through index of query-score pairs is formed and combined with the content index to improve relevance of search results.

Claims

1. A method for indexing content items based on click-through features, the method comprising:

generating a training set comprising one or more query-content item pairs, wherein a given query-content item pair has one or more click-through features associated therewith, the one or more click-through features including two or more of an average amount of time users stay on a website associated with a given URL, a spam score of the given URL, expected clicks at a position of the given URL in a given search results page and frequency of a query in a query log;

labeling one or more query-content item pairs in the training set by assigning click score thereto based on the one or more click-through features thereof, wherein labeling a given content item in the training set comprises providing a given query-content item pair to a human judge to assign a click score;

determining a click score function using a loss function based on the click scores of the labeled query-content item pairs and the click-through features thereof;

applying the click score function to a plurality of unlabeled query-content item pairs to determine click scores thereof based on the one or more click-through features of the unlabeled query-content item pairs;

generating an inverted click-through index of the unlabeled query-content item pairs and the associated query-score pairs, wherein a key to the index is a URL of the content item; and

combining the inverted click-through index with a content index by associating the unlabeled query-content item pairs with content items in the content index.

2. The method of claim 1 further comprising the step of identifying a content item using a Uniform Resource Locator (“

URL”

).

3. The method of claim 1 wherein determining the click score function comprises minimizing a difference between a click score for the given query-content item pair and an output of the click score function.

4. The method of claim 1 wherein the loss function is selected from a set of loss functions including least mean square loss function, a query pairwise loss function and a loss function that compares a desired ranking order with an output ranking order.

5. The method of claim 1 comprising storing the determined click score function for application to an unlabeled query-content item pair.

6. The method of claim 1 comprising the steps of:

retrieving one or more content items in a result set in response to receipt of the query from the user;

for a given content item in the result set, determining a click-through feature vector for the given content item;

applying the click score function to the click-through feature vector for the given content item to generate a click score for the given content item;

generating one or more <

query, content-item, click-score>

data structures for the given content item;

indexing one or more <

query, click-score>

pairs for the content item from the <

query, content-item, click-score>

data structures.

7. A system of one or more processing devices for indexing and searching content items based on its one or more click-through features, the system comprising:

an index component, executed on the one or more processing devices, operative to determine a click score function using a loss function based on a training set of labeled query-content item pairs and the click-through features thereof, the one or more click-through features including two or more of an average amount of time users stay on a website associated with a given URL, a spam score of the given URL, expected clicks at a position of the given URL in a given search results page and frequency of a query in a query log, wherein the training set of labeled query-content items comprises a plurality of query-content items having click scores assigned thereto by a human judge, assign click scores to a plurality of unlabeled query-content item pairs through application of the click score function to the one or more click-through features, generate an inverted click-through index of the unlabeled content items and the associated query-score pairs and combine the inverted click-through index with a content index by associating the unlabeled query-content item pairs with content items in the content index;

a relevance engine, executed on the one or more processing devices, operative to receive one or more query scores for one or more content items and generate one or more relevance scores therefore; and

a search engine, executed on the one or more processing devices, operative to retrieve one or more content items in a result set in response to receipt of the query from the user and order the content items in the result set according to the relevance scores from the relevance engine.

8. The system of claim 7 wherein a Uniform Resource Locator (URL) of a content item is used as a key to locate the given content item the inverted click-through index.

9. The system of claim 7 wherein the click score function is determined by minimizing difference between click score for the given query-content item pair in the training set and the output of the click score function.

10. The system of claim 7 wherein the loss function is selected from a set of loss functions including a least mean square loss function, a query pairwise loss function and a loss function that compares a desired ranking order with an output ranking order.

11. The system of claim 7 wherein the index component is operative to store the determined click score function for application to an unlabeled query-content item pair.