METHOD AND SYSTEM FOR PREDICTING SEARCH RESULTS QUALITY IN VERTICAL RANKING

• US 20160019213A1
• Filed: 07/16/2014
• Published: 01/21/2016
• Est. Priority Date: 07/16/2014
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Carmel, David, Wolff, Ran
• Granted Patent: US 10,146,872 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods, systems and programming for predicting search results quality. In one example, a search query is received from a user. A plurality of search results are obtained from a content source based on the search query. The plurality of search results are ranked based on their relevance scores with respect to the search query. A distribution of the relevance scores of the plurality of search results is normalized in each position of the ranking. A metric of the content source is computed based on the normalized distribution of the relevance scores. The metric indicates a relevance between the plurality of search results and the search query.

Claims

1. A method, implemented on at least one machine each of which has at least one processor, storage, and a communication platform connected to a network for predicting search results quality, the method comprising the steps of:

receiving a search query from a user;

obtaining a plurality of search results from a content source based on the search query, wherein the plurality of search results are ranked based on their relevance scores with respect to the search query;

normalizing a distribution of the relevance scores of the plurality of search results in each position of the ranking; and

computing a metric of the content source based on the normalized distribution of the relevance scores, wherein the metric indicates a relevance between the plurality of search results and the search query.

4. The method of claim 1, wherein normalizing a distribution of the relevance scores comprises:

building an order-statistic model based on a first set of the search results; and

generating estimated relevance scores of a second set of the search results based on the order-statistic model.

5. The method of claim 4, wherein computing a metric includes:

comparing the relevance scores of the second set of the search results with the estimated relevance scores of the second set of the search results.

6. The method of claim 4, wherein the first set of the search results are ranked lower than the second set of the search results.

7. The method of claim 4, wherein the order-statistic model is built based on the relevance scores of the first set of the search results and the positions of the first set of the search results in the ranking.

8. The method of claim 4, wherein the estimated relevance scores of the second set of the search results are generated based on the positions of the second set of the search results in the ranking.

9. The method of claim 4, wherein the relevance scores of the first set of the search results are approximated by a normal distribution.

3-2. The system of claim 12, wherein the mean and the standard deviation of relevance scores in the position are obtained by:

obtaining a plurality of sample query results from the content source based on each of a plurality of sample queries, each of the sample query results being ranked in the position; and

computing the mean and the standard deviation of the plurality of sample queries results in the position.

10. The method of claim 1, wherein the content source includes a vertical in vertical search.

10-1. A system having at least one processor storage, and a communication platform for predicting search results quality, the system comprising:

a search engine configured to receive a search query from a user and obtain a plurality of search results from a content source based on the search query, wherein the plurality of search results are ranked based on their relevance scores with respect to the search query;

a normalization module configured to normalize a distribution of the relevance scores of the plurality of search results in each position of the ranking; and

a ranking module configured to compute a metric of the content source based on the normalized distribution of the relevance scores, wherein the metric indicates a relevance between the plurality of search results and the search query.

12. The system of claim 11, wherein the ranking engine includes a normalization module configured to, in each position of the ranking, compute a normalized relevance score of the respective search result based on a mean and a standard deviation of relevance scores in the position.

2. The method of claim 1, wherein normalizing a distribution of the relevance scores includes:

in each position of the ranking, computing a normalized relevance, score of the respective search result based on a mean and a standard deviation of relevance scores in the position.

3. The method of claim 2, wherein the mean and the standard deviation of relevance scores in the position are obtained by:

obtaining a plurality of sample query results from the content source based on each of a plurality of sample queries, each of the sample query results being ranked in the position; and

computing the mean and the standard deviation of the plurality of sample queries results in the position.

14. The system of claim 11, wherein the ranking engine includes a normalization module configured to

build an order-statistic model based on a first set of the search results;

and

generate estimated relevance scores of a second set of the search results based on the order-statistic model.

15. The system of claim 14, wherein the normalization module is further configured to compare the relevance scores of the second set of the search results with the estimated relevance scores of the second set of the search results.

16. The system of claim 14, wherein the first set of the search results are ranked lower than the second set of the search results.

17. The system of claim 14, wherein the order-statistic model is built based on the relevance scores of the first set of the search results and the positions of the first set of the search results in the ranking.

18. The system of claim 14, wherein the estimated relevance scores of the second set of the search results are generated based on the positions of the second set of the search results in the ranking.

19. The system of claim 14, wherein the relevance scores of the first set of the search results are approximated by a normal distribution.

20. A non-transitory machine-readable medium having information recorded thereon for predicting, search results quality, wherein the information, when read by the machine, causes the machine to perform the following:

receiving a search query from a user;

obtaining a plurality of search results from a content source based on the search query, wherein the plurality of search results are ranked based on their relevance scores with respect to the search query;

normalizing a distribution of the relevance scores of the plurality of search results in each position of the ranking; and

computing a metric of the content source based on the normalized distribution of the relevance scores, wherein the metric indicates a relevance between the plurality of search results and the search query.