Machine translation for query expansion

• US 9,569,527 B2
• Filed: 09/14/2012
• Issued: 02/14/2017
• Est. Priority Date: 06/22/2007
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Riezler, Stefan, Vasserman, Alexander L.
• Primary Examiner(s): WILSON, KIMBERLY LOVEL
• Application Number:
  US13/618,337
• Publication Number:
  US 20130031122A1
• Time in Patent Office: 1,614 Days
• Field of Search: 707/736
• US Class Current: 1/1
• CPC Class Codes: G06F 16/3338 Query expansion

Abstract

Methods, systems and apparatus, including computer program products, for expanding search queries. One method includes receiving a search query, selecting a synonym of a term in the search query based on a context of occurrence of the term in the received search query, the synonym having been derived from statistical machine translation of the term, and expanding the received search query with the synonym and using the expanded search query to search a collection of documents. Alternatively, another method includes receiving a request to search a corpus of documents, the request specifying a search query, using statistical machine translation to translate the specified search query into an expanded search query, the specified search query and the expanded search query being in the same natural language, and in response to the request, using the expanded search query to search a collection of documents.

Claims

1. A computer-implemented method comprising:

identifying a plurality of documents having one or more questions and, for each question, a corresponding answer;

generating a plurality of question-answer pairs from the questions and respective corresponding answers occurring in the plurality of documents;

training a statistical machine translation model using the plurality of question-answer pairs, including using each question of each question-answer pair as a source language input and a corresponding answer of the question-answer pair as a target language input, wherein each question and each corresponding answer are in the same natural language;

translating, using the statistical machine translation model trained on the plurality of question-answer pairs, a phrase into one or more corresponding translated phrases; and

determining one or more synonym pairs by comparing the phrase with the one or more corresponding translated phrases.

2. The method of claim 1, wherein determining one or more synonym pairs by comparing the phrase with the one or more corresponding translated phrases comprises determining that a term in a particular phrase is different from a corresponding translated term in a particular translated phrase.

3. The method of claim 1, wherein determining one or more synonym pairs comprises:

computing a translation likelihood for a phrase and a corresponding translated phrase;

generating a synonym pair from a term in the phrase and a corresponding translated term in the corresponding translated phrase;

assigning a score to the synonym pair, wherein the score is derived from the translation likelihood of the phrase and the corresponding translated phrase; and

determining that the term and the corresponding translated term are synonyms using the assigned score of the synonym pair.

4. The method of claim 3, where each phrase is a search query from a query log.

5. The method of claim 1, further comprising:

receiving an initial search query including one or more terms;

determining that a first term in the search query occurs in a synonym pair of the one or more synonym pairs;

in response to determining that the first term in the search query occurs in a synonym pair of the one or more synonym pairs, generating an expanded search query that includes a second term of the synonym pair;

obtaining search results using the expanded search query to search a collection of documents; and

providing the search results in response to the initial search query.

6. The method of claim 1, further comprising:

extracting question text and corresponding answer text from each document;

adding the question text and answer text to a parallel corpus; and

training the statistical machine translation model using the plurality of question-answer pairs in the parallel corpus.

7. The method of claim 1, further comprising:

training a classifier to identify documents having one or more questions and corresponding answers, wherein identifying the plurality of documents having one or more questions and, for each question, a corresponding answer comprises using the trained classifier to identify documents having one or more questions and corresponding answers.

8. A computer-implemented method comprising:

identifying a plurality of queries in a query log, and for each query, one or more search results associated with the query, wherein each search result identifies a corresponding resource and comprises a search result snippet that includes text from the corresponding resource identified by the search result;

generating a plurality of query-snippet pairs, wherein each pair associates a respective query of the plurality of queries with a particular search result snippet from one of the search results associated with the query in the query log;

training a statistical machine translation model using the plurality of query-snippet pairs, including using each query of each query-snippet pair as a source language input, and a corresponding snippet of the query-snippet pair as a target language input, wherein each query and the search result snippet of each query-snippet pair are in the same natural language;

translating, using the statistical machine translation model trained on the plurality of query-snippet pairs and search result snippet pairs, a phrase into one or more corresponding translated phrases; and

determining one or more synonym pairs including comparing the phrase with the one or more corresponding translated phrases.

9. The method of claim 8, wherein determining one or more synonym pairs by comparing the phrase with the one or more corresponding translated phrases comprises determining that a term in a particular phrase is different from a corresponding translated term in a particular translated phrase.

10. The method of claim 8, wherein determining one or more synonym pairs comprises:

computing a translation likelihood for a phrase and a corresponding translated phrase;

generating a synonym pair from a term in the phrase and a corresponding translated term in the corresponding translated phrase;

assigning a score to the synonym pair, wherein the score is derived from the translation likelihood of the phrase and the corresponding translated phrase; and

determining that the term and the corresponding translated term are synonyms using the assigned score of the synonym pair.

11. The method of claim 10, where the phrase is a search query from the query log.

12. The method of claim 8, further comprising:

adding the query-snippet pairs to a parallel corpus; and

training the statistical machine translation model using the plurality of query-snippet pairs in the parallel corpus.

13. The method of claim 8, further comprising:

receiving an initial search query including one or more terms;

determining that a first term in the search query occurs in a synonym pair of the one or more synonym pairs;

in response to determining that the first term in the search query occurs in a synonym pair of the one or more synonym pairs, generating an expanded search query that includes a second term of the synonym pair;

obtaining search results using the expanded search query to search a collection of documents; and

providing the search results in response to the initial search query.

14. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

identifying a plurality of documents having one or more questions and, for each question, a corresponding answer;

generating a plurality of question-answer pairs from the questions and respective corresponding answers occurring in the plurality of documents;

training a statistical machine translation model using the plurality of question-answer pairs, including using each question of each question-answer pair as a source language input and a corresponding answer of the question-answer pair as a target language input, wherein each question and each corresponding answer are in the same natural language;

translating, using the statistical machine translation model trained on the plurality of question-answer pairs, a phrase into one or more corresponding translated phrases; and

determining one or more synonym pairs by comparing the phrase with the one or more corresponding translated phrases.

15. The system of claim 14, wherein determining one or more synonym pairs comprises:

computing a translation likelihood for a phrase and a corresponding translated phrase;

generating a synonym pair from a term in the phrase and a corresponding translated term in the corresponding translated phrase;

assigning a score to the synonym pair, wherein the score is derived from the translation likelihood of the phrase and the corresponding translated phrase; and

determining that the term and the corresponding translated term are synonyms using the assigned score of the synonym pair.

16. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

identifying a plurality of queries in a query log, and for each query, one or more search results associated with the query, wherein each search result identifies a corresponding resource and comprises a search result snippet that includes text from the corresponding resource identified by the search result;

generating a plurality of query-snippet pairs, wherein each pair associates a respective query of the plurality of queries with a particular search result snippet from one of the search results associated with the query in the query log;

training a statistical machine translation model using the plurality of query-snippet pairs, including using each query of each query-snippet pair as a source language input, and a corresponding snippet of the query-snippet pair as a target language input, wherein each query and the search result snippet of each query-snippet pair are in the same natural language;

translating, using the statistical machine translation model trained on the plurality of query-snippet pairs, a phrase into one or more corresponding translated phrases; and

determining one or more synonym pairs including comparing the phrase with the one or more corresponding translated phrases.

17. The system of claim 16, wherein determining one or more synonym pairs comprises:

computing a translation likelihood for a phrase and a corresponding translated phrase;

generating a synonym pair from a term in the phrase and a corresponding translated term in the corresponding translated phrase;

assigning a score to the synonym pair, wherein the score is derived from the translation likelihood of the phrase and the corresponding translated phrase; and

determining that the term and the corresponding translated term are synonyms using the assigned score of the synonym pair.