Estimating confidence for query revision models

• US 9,697,249 B1
• Filed: 06/12/2015
• Issued: 07/04/2017
• Est. Priority Date: 09/30/2003
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Gomes, Benedict A., Nayak, P. Pandurang, Bailey, David R., Battle, Alexis J.
• Primary Examiner(s): Meng, Jau-Shya
• Application Number:
  US14/737,720
• Time in Patent Office: 753 Days
• Field of Search: None
• CPC Class Codes: G06F 16/242 Query formulation
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/3322 using system suggestions G06F16/3325 takes precedence
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

An information retrieval system includes a query revision architecture that integrates multiple different query revisers, each implementing one or more query revision strategies. A revision server receives a user'"'"'s query, and interfaces with the various query revisers, each of which generates one or more potential revised queries. The revision server evaluates the potential revised queries, and selects one or more of them to provide to the user. A session-based reviser suggests one or more revised queries, given a first query, by calculating an expected utility for the revised query. The expected utility is calculated as the product of a frequency of occurrence of the query pair and an increase in quality of the revised query over the first query.

Claims

1. A computer-implemented method comprising:

obtaining training data including;

for each of one or more original queries;

(i) one or more features of the original query,

(ii) one or more features of a revised query which was generated by a query revisor for the original query, and

(iii) a label indicating whether the revised query is a successful revision of the original query;

training, using the training data obtained for each of the one or more original queries that includes, for each respective original query, (i) the one or more features of the original query, (ii) the one or more features of the revised query which was generated by the query revisor for the original query, and (iii) the label indicating whether the revised query is a successful revision of the original query, a predictive model for predicting a likelihood that a given revised query is a successful revision of a given original query; and

after receiving a particular original query, (a) using the predictive model to predict a likelihood that a particular revised query is a successful revision of the particular original query, and (b) determining whether to submit the particular revised search query to a search engine based on the likelihood.

2. The method of claim 1, wherein the likelihood that a given revised query is a successful revision of a given original query comprises a likelihood that a user remains on a clicked-through page associated with a result of the given revised query that is obtained by the query revisor for the given original query.

3. The method of claim 1, wherein the one or more features of a revised query which was generated by a query revisor for the original query include each word of each revised query.

4. The method of claim 1, wherein the one or more features of a revised query which was generated by a query revisor for the original query include an identification of one or more revision techniques that the query revisor used to produce the revised query.

5. The method of claim 1, wherein the one or more features of a revised query which was generated by a query revisor for the original query include a length of a click on a link associated with each revised query.

6. The method of claim 1, wherein the one or more features of the original query include each word of each original query.

7. The method of claim 1, wherein the one or more features of the original query include a length of each original query.

8. The method of claim 1, wherein the one or more features of the original query include a topic cluster associated with each original query.

9. The method of claim 1, wherein the one or more features of the original query include an information retrieval score associated with each original query.

10. The method of claim 1, wherein the one or more features of the original query include a quantity of search results associated with each original query.

11. The method of claim 1, wherein:

the particular original query is not included among the one or more original queries, and

the particular revised query is not included among the revised queries which were obtained for each of the one or more original queries.

12. The method of claim 1, comprising ranking each particular revised query based on the respective predicted likelihood.

13. The method of claim 12, comprising:

selecting a top n of the ranked, particular revised queries; and

providing only the top n of the ranked, particular revised queries for output.

14. The method of claim 1, wherein the respective predicted likelihood for each particular revised query is obtained after the particular original query is received.

15. The method of claim 1, comprising:

generating (i) one or more features of the particular original query, and (ii) one or more features from the particular revised query for the particular original query.

16. A system comprising:

one or more computers; and

a non-transitory computer-readable medium coupled to the one or more computers having instructions stored thereon, which, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

obtaining training data including, for each of one or more original queries;

(i) one or more features of the original query,

(ii) one or more features of a revised query which was generated by a query revisor for the original query, and

(iii) a label indicating whether the revised query is a successful revision of the original query;

training, using the training data obtained for each of the one or more original queries that includes, for each respective original query, (i) the one or more features of the original query, (ii) the one or more features of the revised query which was generated by the query revisor for the original query, and (iii) the label indicating whether the revised query is a successful revision of the original query, a predictive model for predicting a likelihood that a given revised query is a successful revision of a given original query; and

after receiving a particular original query, (a) using the predictive model to predict a likelihood that a particular revised query is a successful revision of the particular original query, and (b) determining whether to submit the particular revised search query to a search engine based on the likelihood.

17. The system of claim 16, wherein the likelihood that a given revised query is a successful revision of a given original query comprises a likelihood that a user remains on a clicked-through page associated with a result of the given revised query that is obtained by the query revisor for the given original query.

18. The system of claim 16, wherein the one or more features of a revised query which was generated by a query revisor for the original query include each word of each revised query.

19. A non-transitory computer-readable storage medium encoded with a computer program, the program comprising instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising:

obtaining training data including, for each of one or more original queries;

(i) one or more features of the original query,

(ii) one or more features of a revised query which was generated by a query revisor for the original query, and

(iii) a label indicating whether the revised query is a successful revision of the original query;

training, using the training data obtained for each of the one or more original queries that includes, for each respective original query, (i) the one or more features of the original query, (ii) the one or more features of the revised query which was generated by the query revisor for the original query, and (iii) the label indicating whether the revised query is a successful revision of the original query, a predictive model for predicting a likelihood that a given revised query is a successful revision of a given original query; and

after receiving a particular original query, (a) using the predictive model to predict a likelihood that a particular revised query is a successful revision of the particular original query, and (b) determining whether to submit the particular revised search query to a search engine based on the likelihood.

20. The non-transitory computer-readable storage medium of claim 19, wherein the likelihood that a given revised query is a successful revision of a given original query comprises a likelihood that a user remains on a clicked-through page associated with a result of the given revised query that is obtained by the query revisor for the given original query.