Reranking QA answers using language modeling

• US 20080040114A1
• Filed: 08/11/2006
• Published: 02/14/2008
• Est. Priority Date: 08/11/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhou, Ming, Chen, Yi
• Granted Patent: US 7,856,350 B2
• Time in Patent Office: Days
• US Class Current: 704/257
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning

Abstract

In a QA (Question/Answer) system, candidate answers in response to a question received are ranked by probabilities estimated by a language model. The language model is created based on an ordered centroid created from the question and information learned from an information source such as the Internet.

Claims

1. A computer implemented method for reranking candidate answers related to a question in a QA system comprising:

receiving a question and determining a question focus of the question;

querying a source of information with the question focus and obtaining one or more relevant documents;

obtaining candidate answers from said one or more relevant documents; and

reranking the candidate answers using a language model related to the question.

2. The computer implemented method of claim 1 and further comprising removing redundancies from the candidate answers.

3. The computer implemented method of claim 1 wherein the question is a definitional question.

4. The computer implemented method of claim 1 wherein the question is a factoid.

5. The computer implemented method of claim 1 wherein the language model is an n-gram model.

6. The computer implemented method of claim 5 wherein the n-gram model is a bi-gram model.

7. The computer implemented method of claim 5 wherein the n-gram model is a bi-term model.

8. The computer implemented method of claim 1 and further comprising, prior to reranking the candidate answers, building the language model.

9. The computer implemented method of claim 8 wherein building the language model comprises:

building the language model using co-occurring terms with the question focus.

10. The computer implemented method of claim 9 wherein building the language model comprises:

obtaining relevant sentences and/or phrases having the question focus and one or more co-occurring terms.

11. The computer implemented method of claim 10 wherein obtaining relevant sentences comprises querying a second source of information.

12. The computer implemented method of claim 11 wherein the second source of information comprises the Internet.

13. A computer implemented method for building a language model for use in a QA system receiving definitional questions:

receiving a definitional question and determining a question focus of the definitional question;

obtaining information relevant to the question focus;

generating a centroid vector based on the question focus and the information relevant to the question focus;

generating an ordered centroid based on centroid vector; and

building a language model based on the ordered centroid.

14. The computer implemented method of claim 13 wherein obtaining information includes searching a source of information using a search engine and a query comprising the question focus and clue words based on a type of the definitional question.

15. The computer implemented method of claim 14 wherein obtaining information includes:

receiving results based on the query comprising the question focus and clue words based on the type of the definitional question;

processing the results to obtain expansion terms; and

querying the source of information with the question focus and selected expansion terms based on the results; and

wherein generating the centroid vector comprises using the results from querying the source of information with the question focus and selected expansion terms based on the results.

16. The computer implemented method of claim 15 wherein generating the centroid vector includes generating the centroid vector based on co-occurring terms proximate the question focus in phrases and/or sentences in the results.

17. The computer implemented method of claim 13 and using the language model to rerank candidate answers in response to a definitional question in a QA system.

18. A computer readable medium having instructions, which when executed by a computer, implement a QA system that receives a definitional question and provides an answer thereto, the instructions comprising:

receiving a definitional question and determining a question focus of the definitional question;

querying a source of information with the question focus and obtaining one or more relevant documents;

separating the one or more documents into sentences or phrases and retaining sentences or phrases answers having the question focus as candidate answers; and

reranking the candidate answers using a language model based on the definitional question, the language model providing an indication of term dependence of terms in the candidate answers.

19. The computer readable medium of claim 18 and wherein the instructions further comprise removing redundancies from the candidate answers.

20. The computer readable medium of claim 18 wherein the language model is n-gram model.