Learning-Based Processing of Natural Language Questions

• US 20140006012A1
• Filed: 07/02/2012
• Published: 01/02/2014
• Est. Priority Date: 07/02/2012
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhou, Ming, Liu, Xiaohua, Shum, Heung-Yeung, Wei, Furu, Sun, Hong, Duan, Yajuan, Sun, Chengjie
• Application Number:
  US13/539,674
• Publication Number:
  US 20140006012A1
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 40/295 Named entity recognition

Abstract

Techniques described enable answering a natural language question using machine learning-based methods to gather and analyze evidence from web searches. A received natural language question is analyzed to extract query units and to determine a question type, answer type, and/or lexical answer type using rules-based heuristics and/or machine learning trained classifiers. Query generation templates are employed to generate a plurality of ranked queries to be used to gather evidence to determine the answer to the natural language question. Candidate answers are extracted from the results based on the answer type and/or lexical answer type, and ranked using a ranker previously trained offline. Confidence levels are calculated for the candidate answers and top answer(s) may be provided to the user if the confidence levels of the top answer(s) surpass a threshold.

Claims

1. A computer-implemented method comprising:

analyzing a natural language question to predict a question type and an answer type for the natural language question;

formulating a ranked plurality of search queries based at least partly on the question type and on one or more query units extracted from the natural language question;

determining one or more candidate answers from a plurality of search results resulting from execution of at least some of the ranked plurality of search queries by a search engine, the determining based at least partly on the answer type;

ranking the one or more candidate answers according to a confidence level determined for each of the one or more candidate answers; and

providing a highest-ranked candidate answer of the one or more candidate answers based at least partly on a determination that the highest-ranked candidate answer has a confidence level higher than a predetermined threshold confidence.

2. The method of claim 1 wherein the question type is predicted through use of a classifier that is trained using a machine learning technique with multiple features.

3. The method of claim 2 wherein the machine learning technique is a support vector machine (SVM) technique.

4. The method of claim 1 wherein the answer type is predicted based at least partly on a plurality of predefined rules.

5. The method of claim 1 further comprising:

employing a ranker to rank the plurality of search queries, the ranker trained using a machine learning technique; and

determining a highest-ranked number of the plurality of search queries for execution by the search engine.

6. The method of claim 1 further comprising:

filtering the plurality of search results to remove at least one of a duplicate search result or a noise search result, prior to determining the one or more candidate answers.

7. The method of claim 1 wherein determining the one or more candidate answers includes:

extracting one or more named entities from the plurality of search results, the one or more named entities corresponding to the answer type, the extracting based at least partly on a dictionary matching of the one or more named entities with text of the plurality of search results; and

normalizing the one or more named entities to determine the one or more candidate answers.

8. The method of claim 1 wherein the one or more candidate answers are ranked through use of a ranker that is trained using a machine learning technique.

9. A system comprising:

at least one memory;

at least one processor in communication with the at least one memory; and

a natural language question processing component stored in the at least one memory and executed by the at least one processor to;

analyze a received natural language question to determine a question type and an answer type for the natural language question;

determine one or more query units from the natural language question;

formulate a plurality of search queries based at least partly on the question type and the one or more query units;

determine one or more candidate answers from a plurality of search results based at least partly on the answer type, the plurality of search results resulting from execution of at least some of the plurality of search queries by a search engine; and

rank the one or more candidate answers based at least partly on a confidence level determined for each of the one or more candidate answers.

10. The system of claim 9 wherein the question type is at least one of a factoid type, a definition type, a puzzle type, or a math type.

11. The system of claim 9 wherein the answer type is at least one of a person, a location, a date, a time, a quantity, an event, an organism, an object, or a concept.

12. The system of claim 9 wherein the natural language question processing component further operates to determine a lexical answer type for the natural language question based on the analysis of the natural language question, wherein the one or more candidate answers are determined further based at least partly on the lexical answer type.

13. The system of claim 12 wherein the lexical answer type is a subset of the answer type.

14. The system of claim 9 wherein 1 wherein determining the one or more query units is based at least partly on a grammar-based analysis of the natural language question.

15. The system of claim 9 wherein the one or more query units includes at least one of a word, a noun-phrase, a named entity, a quotation, a fact, a syntactic structure, or a paraphrase.

16. The system of claim 9 further comprising:

a machine learning component stored in the at least one memory and executed by the at least one processors to train a ranker using a machine learning technique;

wherein the natural language question processing component further operates to;

rank the plurality of search queries using the ranker; and

determine a highest-ranked number of the plurality of search queries for execution by the search engine.

17. One or more computer-readable storage media storing instructions that, when executed by at least one processor, instruct the at least one processor to perform actions comprising:

analyzing a received natural language question to determine a question type and an answer type for the natural language question;

formulating a plurality of search queries based at least partly on the question type and on one or more query units extracted from the natural language question;

extracting one or more candidate answers from a plurality of search results resulting from execution of at least some of the plurality of search queries; and

ranking the one or more candidate answers according to a confidence level determined for each of the one or more candidate answers.

18. The one or more computer-readable storage media of claim 17 wherein the actions further comprise:

providing a highest-ranked candidate answer based at least partly on a determination that the confidence level of the highest-ranked candidate answer is greater than a predetermined threshold confidence.

19. The one or more computer-readable storage media of claim 17 wherein each of the plurality of search results includes an address for a web site and a snippet of content from the web site.

20. The one or more computer-readable storage media of claim 17 wherein ranking the one or more candidate answers is based on a weight vector that is trained using a machine learning technique.