System and method for providing answers to questions

• US 9,703,861 B2
• Filed: 05/21/2014
• Issued: 07/11/2017
• Est. Priority Date: 05/14/2008
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Zadrozny, Wlodek W., Brown, Eric W., Ferrucci, David, Lally, Adam
• Primary Examiner(s): Chbouki, Tarek
• Application Number:
  US14/283,929
• Publication Number:
  US 20140258286A1
• Time in Patent Office: 1,147 Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

System and method for providing answers to questions based on any corpus of data implements a method that generates a number of candidate passages from the corpus that answer an input query, and finds the correct resulting answer by collecting supporting evidence from the multiple passages. By analyzing all retrieved passages and that passage'"'"'s metadata in parallel, an output plurality of data structures is generated including candidate answers based upon the analyzing. Then, supporting passage retrieval operations are performed upon the set of candidate answers, and for each candidate answer, the data corpus is traversed to find those passages having candidate answer in addition to query terms. All candidate answers are automatically scored by a plurality of scoring modules, each producing a module score. The modules scores are processed to determine one or more query answers; and, a query response is generated based on the one or more query answers.

Claims

1. A computer implemented method of generating answers to natural language questions, said method comprising:

receiving said natural language question;

processing said question using a plurality of natural language processing techniques to obtain a plurality of searchable components;

conducting a first search in a corpus based on said plurality of searchable components to obtain a plurality of relevant documents;

analyzing said plurality documents to generate a plurality of candidate answers;

after said candidate answers are generated, retrieving a plurality of supporting passages from said corpus based on said candidate answers;

analyzing said candidate answers based on grammatical and semantic structures in said query and in said retrieved supporting passages;

generating a confidence score for each analyzed candidate answer, said confidence score based on a ratio of the number of query terms present in a supporting passage with a total number of the searchable components of the received question;

and outputting said candidate answers and associated confidence scores.

2. The method as claimed in claim 1, wherein said processing said question further comprises:

determining one or more predicate argument structures of said query.

3. The method as claimed in claim 1, wherein said processing said question further comprises:

determining one or more lexical answer types for each input query.

4. The method as claimed in claim 1, wherein said processing said question further comprises:

extending one or more searchable components using a function for term weighting and query expansion.

5. The method as claimed in claim 1, wherein said retrieving said plurality of supporting passages and generating a confidence score for each candidate answer based on said supporting passages occurs in parallel operations.

6. The method as claimed in claim 5, wherein said confidence score comprises a term match score, said method further comprising:

counting a number of query terms in a passage; and

determining if said number matches a number of terms in a candidate answer.

7. The method as claimed in claim 5, wherein said confidence score comprises a textual alignment score, said method comprising:

determining if a placement of words in said passages are in alignment with a placement of words of said candidate answers.

8. The method as claimed in claim 7, wherein said determining if placement of words in said supporting passages are in alignment includes:

determining whether said words in said passages are one of;

a same order, a similar order, or with a similar distance between them.

9. The method as claimed in claim 5, wherein said confidence score comprises a deeper analysis score, said method further comprising:

determining a meaning of the passage and the input query; and

computing a degree of satisfying a lexical or a semantic relation in the passage.

10. The method as claimed in claim 1, further comprising:

ranking said candidate answers based on their scores to determine one or more query answers, wherein an output candidate answer includes one of;

a single query answer, or a ranked list of query answers.

11. The method as claimed in claim 10, further comprising:

generating an elaboration question for delivery to a user, and

receiving information from a user responsive to said elaboration question for use in determining said query answer.

12. The method as claimed in claim 10, further comprising:

determining if a query answer or ranked list of query answers is above a threshold rank level, and if below said threshold rank level,

delivering a response to a user comprising one or more clarification questions, and

receiving information from a user responsive to said one or more clarification questions, said received user information being added to said query.

13. The method as claimed in claim 10, further comprising:

producing, using machine learning techniques, a trained model component from prior data, said prior data encoding on one or more of;

features of candidate answers, features of passages having the candidate answers, the candidate answer scores and whether the candidate answer was correct or not; and

utilizing said trained model component for ranking said candidate answers.

14. The method as claimed in claim 13, further comprising:

producing a prediction function from said trained model component; and

applying said prediction function for answer ranking, said applying implementing methods to weight said candidate answers.

15. The method as claimed in claim 10, further comprising:

determining a final candidate answer by;

collecting results across all supporting passages having said scored candidate answers,

normalizing and merging candidate answers produced by a same answer scorer across multiple instances of the candidate answer, and aggregating the results; and

,

applying a candidate answer scoring function to produce said final candidate answer.

16. The method as claimed in claim 15, wherein said applying a candidate answer scoring function comprises one of:

performing context independent scoring where the answer is scored independently of the supporting passage;

or

performing context dependent scoring where the answer score depends on the supporting passage content.

17. A system for generating answers to natural language questions comprising:

a memory storage device;

a hardware processor in communication with said memory storage device and configured to;

receive said natural language question;

process said question using a plurality of natural language processing techniques to obtain a plurality of searchable components;

conduct a first search in a corpus based on said plurality of searchable components to obtain a plurality of relevant documents;

analyze said plurality documents to generate a plurality of candidate answers;

after said candidate answers are generated, retrieve a plurality of supporting passages from said corpus based on said candidate answers;

analyzing said candidate answers based on grammatical and semantic structures in said query and in said supporting passages;

generate a confidence score for each analyzed candidate answer, said confidence score based on a ratio of the number of query terms present in a supporting passage with a total number of the searchable components of the received question; and

output said candidate answers and associated confidence scores.

18. The system as claimed in claim 17, wherein said natural language question and said output candidate answers is provided in accordance with one or more of multiple modalities including text, audio, image, video, tactile or gesture.

19. The system as claimed in claim 18, wherein said hardware processor is further configured to:

determine, from said natural language question, one or more predicate argument structures; and

,

determine, from said natural language question, one or more lexical answer types.

20. The system as claimed in claim 18, wherein said retrieving said plurality of supporting passages and generating a confidence score for each candidate answer based on said supporting passages occurs in parallel operations.

21. The system as claimed in claim 20, wherein said hardware processor is further configured to conduct, in parallel, one or more analyses in a plurality of scoring modules to each produce a confidence score, wherein said confidence score comprises one or more of:

conduct, in parallel, one or more analyses in each of a plurality of scoring modules to each produce a score, wherein said score comprises one or more of;

a term match score obtained by counting a number of query terms in a passage; and

determining if said number matches a number of terms in a candidate answer, or

a textual alignment score obtained by determining if a placement of words in said passages are in alignment with a placement of words of said candidate answers, or

a deeper analysis score obtained by determining a meaning of the passage and the input query; and

computing a degree of satisfying a lexical or a semantic relation in the passage.

22. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform method steps for generating answers to natural language questions, said method steps including the steps of:

receiving said natural language question;

processing said question using a plurality of natural language processing techniques to obtain a plurality of searchable components;

conducting a first search in a corpus based on said plurality of searchable components to obtain a plurality of relevant documents;

analyzing said plurality documents to generate a plurality of candidate answers;

after said candidate answers are generated, retrieving a plurality of supporting passages from said corpus based on said candidate answers;

analyzing said candidate answers based on grammatical and semantic structures in said query and in said supporting passages;

generating a confidence score for each analyzed candidate answer, said confidence score based on a ratio of the number of query terms present in a supporting passage with a total number of the searchable components of the received question;

and outputting said candidate answers and associated confidence scores.

23. The program storage device as claimed in claim 22, wherein said method further comprises:

determining, from said natural language question, one or more predicate argument structures for each input query; and

,

determining, from said natural language question, one or more lexical answer types for each input query.

24. The program storage device as claimed in claim 23, wherein said retrieving said plurality of supporting passages and generating a confidence score for each candidate answer based on said supporting passages occurs in parallel operations.

25. The program storage device as claimed in claim 24, wherein said method further comprises:

conducting, in parallel, one or more analyses in each of a plurality of scoring modules to each produce a score, wherein said score comprises one or more of;

a term match score obtained by counting a number of query terms in a passage; and

determining if said number matches a number of terms in a candidate answer, or

a textual alignment score obtained by determining if a placement of words in said passages are in alignment with a placement of words of said candidate answers, or

a deeper analysis score obtained by determining a meaning of the passage and the input query; and

computing a degree of satisfying a lexical or a semantic relation in the passage.