Prediction of human ratings or rankings of information retrieval quality
First Claim
Patent Images
1. A method performed by one or more server devices, the method comprising:
- determining, by at least one of the one or more server devices, search results associated with search queries,the search queries and the search results forming respective search query/search result pairs;
determining, by at least one of the one or more server devices and for each of the search query/search result pairs, signals relating to the search query/search result pair,the signals, for a particular search query/search result pair of the search query/search result pairs, including;
a first signal defining a time duration for selection of the search result, of the particular search query/search result pair, from search results identified for the search query of the particular search query/search result pair,the time duration being based on a time between when the search results are presented for the search query, of the particular search query/search result pair, and when the search result, of the particular search query/search result pair, is selected, andat least one of;
a second signal indicating selection of the search result, of the particular search query/search result pair, before selection of any other ones of the search results for the search query of the particular search query/search result pair, ora third signal indicating selection of another search result, of the search results identified for the search query of the particular search query/search result pair, before selection of the search result of the particular search query/search result pair,the first signal, the second signal, and the third signal being based on information, in one or more click logs, regarding user selections of particular search results,the particular search results being identified based on particular search queries;
training, by at least one of the one or more server devices and based on the signals, a search engine to predict relevance ratings for search results;
identifying, using the search engine, a particular search result that is relevant to a particular search query; and
predicting, using the search engine, a relevance rating for the particular search result.
1 Assignment
0 Petitions
Accused Products
Abstract
A statistical model may be created that relates human ratings of documents to objective signals generated from the documents, search queries, and/or other information (e.g., query logs). The model can then be used to predict human ratings/rankings for new documents/search query pairs. These predicted ratings can be used to, for example, refine rankings from a search engine or assist in evaluating or monitoring the efficacy of a search engine system.
24 Citations
20 Claims
-
1. A method performed by one or more server devices, the method comprising:
-
determining, by at least one of the one or more server devices, search results associated with search queries, the search queries and the search results forming respective search query/search result pairs; determining, by at least one of the one or more server devices and for each of the search query/search result pairs, signals relating to the search query/search result pair, the signals, for a particular search query/search result pair of the search query/search result pairs, including; a first signal defining a time duration for selection of the search result, of the particular search query/search result pair, from search results identified for the search query of the particular search query/search result pair, the time duration being based on a time between when the search results are presented for the search query, of the particular search query/search result pair, and when the search result, of the particular search query/search result pair, is selected, and at least one of; a second signal indicating selection of the search result, of the particular search query/search result pair, before selection of any other ones of the search results for the search query of the particular search query/search result pair, or a third signal indicating selection of another search result, of the search results identified for the search query of the particular search query/search result pair, before selection of the search result of the particular search query/search result pair, the first signal, the second signal, and the third signal being based on information, in one or more click logs, regarding user selections of particular search results, the particular search results being identified based on particular search queries; training, by at least one of the one or more server devices and based on the signals, a search engine to predict relevance ratings for search results; identifying, using the search engine, a particular search result that is relevant to a particular search query; and predicting, using the search engine, a relevance rating for the particular search result. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. One or more server devices comprising:
-
one or more memories to store instructions; and one or more processors to execute the instructions to; determine signals relating to a search query and a search result, the signals including; a first signal defining a time duration for selection of the search result, from search results identified for the search query, based on a particular location at which the search result is presented within the search results,
the time duration being based on a time between when the search results are presented for the search query and when the search result is selected from the particular location within the search results, andat least one of;
a second signal indicating selection of the search result before selection of another search result of the search results identified for the search query, or
a third signal indicating selection of another search result, of the search results identified for the search query, before selection of the search result,the first signal, the second signal, and the third signal being based on information, in one or more click logs, regarding user selections of particular search results, the particular search results being identified based on search queries, train, based on the signals, a search engine to predict relevance ratings for search results of one or more search queries, identify a particular search result that is relevant to a particular search query, and predict, using the search engine, a relevance rating for the particular search result. - View Dependent Claims (9, 10, 11, 12, 13, 14)
-
-
15. A non-transitory computer-readable medium storing instructions, the instructions comprising:
-
one or more instructions which, when executed by one or more processors, cause the one or more processors to determine signals relating to a search query and a search result identified for the search query, the signals including; first information defining a particular sequence of selections relating to the search result and other search results identified for the search query, and at least one of; second information defining time durations between the selections, or third information defining a time duration of the search result being accessed,
the time duration of the search result being accessed being based on the time durations between the selections,the first information, the second information, and the third information being based on information, in one or more click logs, regarding user selections of particular search results, the particular search results being identified based on search queries; one or more instructions which, when executed by the one or more processors, cause the one or more processors to train, based on the signals, a search engine to predict relevance ratings for search results of one or more search queries; one or more instructions which, when executed by the one or more processors, cause the one or more processors to identify a particular search result that is relevant to a particular search query; and one or more instructions which, when executed by the one or more processors, cause the one or more processors to predict, using the search engine, a relevance rating for the particular search result. - View Dependent Claims (16, 17, 18, 19, 20)
-
Specification