Method and system for predicting search results quality in vertical ranking

• US 10,146,872 B2
• Filed: 07/16/2014
• Issued: 12/04/2018
• Est. Priority Date: 07/16/2014
• Status: Expired due to Fees

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Excalibur IP, LLC (Acacia Research Corporation)
• Inventors: Carmel, David, Wolff, Ran
• Primary Examiner(s): Fan, Shiow-Jy
• Application Number:
  US14/332,501
• Publication Number:
  US 20160019213A1
• Time in Patent Office: 1,602 Days
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Methods, systems and programming for predicting search results quality. In one example, a search query is received from a user. A plurality of search results are obtained from a content source based on the search query. The plurality of search results are ranked based on their relevance scores with respect to the search query. A distribution of the relevance scores of the plurality of search results is normalized in each position of the ranking. A metric of the content source is computed based on the normalized distribution of the relevance scores. The metric indicates a relevance between the plurality of search results and the search query.

Claims

1. A method, implemented on at least one machine each of which has at least one processor, storage, and a communication platform connected to a network for predicting search results quality, the method comprising the steps of:

receiving, via the at least one processor, a search query from a user;

obtaining, via the at least one processor, a plurality of search results from each of a plurality of content sources based on the search query, wherein the plurality of search results from each content source is ranked based on their relevance scores with respect to the search query;

normalizing, via the at least one processor, a distribution of the relevance scores of the plurality of search results for each of the plurality of content sources in each position of the ranking by building an order-statistic model based on a first set of the plurality of search results from the each content source and by generating estimated relevance scores of a second set of the plurality of search results from the each content source based on the order-statistic model, wherein the first set is different from the second set;

computing, via the at least one processor, a metric for each of the plurality of content sources based on the normalized distribution of the relevance scores, wherein the metric indicates a relevance between the respective plurality of search results from the content source and the search query;

ranking, via the at least one processor, the plurality of content sources based on the metrics associated with the plurality of content sources;

identifying, via the at least one processor, one or more search results from at least one content source that has a higher ranking; and

providing, via the at least one processor, the one or more search results to the user as a response to the search query.

2. The method of claim 1, wherein computing a metric includes:

comparing the relevance scores of the respective plurality of search results in the second set with the estimated relevance scores of the second set.

3. The method of claim 1, wherein the plurality of content sources are vertical content sources and/or respond to vertical searches.

4. The method of claim 1, wherein the order-statistic model is built based on the relevance scores of the first set and the positions of the first set in the ranking.

5. The method of claim 1, wherein the estimated relevance scores of the second set is generated based on the positions of the second set in the ranking.

6. The method of claim 1, wherein the relevance scores of the first set is approximated by a normal distribution.

7. The method of claim 1, wherein content of a particular topic, media type, or genre are provided in the plurality of search results from a respective content source.

8. A system for predicting search results quality, the system comprising:

at least one processor configured by machine-readable instructions to;

receive a search query from a user;

obtain a plurality of search results from each of a plurality of content sources based on the search query, wherein the plurality of search results from each content source is ranked based on their relevance scores with respect to the search query;

normalize a distribution of the relevance scores of the plurality of search results for each of the plurality of content sources in each position of the ranking by building an order-statistic model based on a first set of the plurality of search results from the each content source and by generating estimated relevance scores of a second set of the plurality of search results from the each content source based on the order-statistic model, wherein the first set is different from the second set;

compute a metric for each of the plurality of content sources based on the normalized distribution of the relevance scores, wherein the metric indicates a relevance between the respective plurality of search results from the content source and the search query,

rank the plurality of content sources based on the metrics associated with the plurality of content sources,

identify one or more search results from at least one content source that has a higher ranking, and

provide the one or more search results to the user as a response to the search query.

9. The system of claim 8, wherein the at least one processor is further configured to compare the relevance scores of the respective plurality of search results in the second set with the estimated relevance scores of the second set.

10. The system of claim 8, wherein the plurality of content sources are vertical content sources and/or respond to vertical searches.

11. The system of claim 8, wherein the order-statistic model is built based on the relevance scores of the first set and the positions of the first set in the ranking.

12. The system of claim 8, wherein the estimated relevance scores of the second set is generated based on the positions of the second set in the ranking.

13. The system of claim 8, wherein the relevance scores of the first set is approximated by a normal distribution.

14. A non-transitory machine-readable medium having information recorded thereon for predicting search results quality, wherein the information when read by at least one processor, causes the at least one processor to perform the following:

receiving a search query from a user;

obtaining a plurality of search results from each of a plurality of content sources based on the search query, wherein the plurality of search results from each content source is ranked based on their relevance scores with respect to the search query;

normalizing a distribution of the relevance scores of the plurality of search results for each of the plurality of content sources in each position of the ranking by building an order-statistic model based on a first set of the plurality of search results from the each content source and by generating estimated relevance scores of a second set of the plurality of search results from the each content source based on the order-statistic model, wherein the first set is different from the second set;

computing a metric for each of the plurality of content sources based on the normalized distribution of the relevance scores, wherein the metric indicates a relevance between the respective plurality of search results from the content source and the search query;

ranking the plurality of content sources based on the metrics associated with the plurality of content sources;

identifying one or more search results from at least one content source that has a higher ranking; and

providing the one or more search results to the user as a response to the search query.

15. A method, implemented on at least one machine each of which has at least one processor, storage, and a communication platform connected to a network for predicting search results quality, the method comprising the steps of:

receiving, via the at least one processor, a search query from a user;

obtaining, via the at least one processor, a plurality of search results from each of a plurality of content sources based on the search query, wherein the plurality of search results from each content source is ranked based on their relevance scores with respect to the search query;

normalizing, via the at least one processor, a distribution of the relevance scores of the plurality of search results for each of the plurality of content sources in each position of the ranking by computing, in the each position, a normalized relevance score of the respective search result based on a mean and a standard deviation of relevance scores in the position that are obtained by obtaining a plurality of sample query results from the plurality of content sources based on each of a plurality of sample queries, each of the sample query results being ranked in the position, and by computing the mean and the standard deviation of the plurality of sample queries results in the position;

computing, via the at least one processor, a metric for each of the plurality of content sources based on the normalized distribution of the relevance scores, wherein the metric indicates a relevance between the respective plurality of search results from the content source and the search query;

ranking, via the at least one processor, the plurality of content sources based on the metrics associated with the plurality of content sources;

identifying, via the at least one processor, one or more search results from at least one content source that has a higher ranking; and

providing, via the at least one processor, the one or more search results to the user as a response to the search query.