Clarification of submitted questions in a question and answer system

• US 9,361,386 B2
• Filed: 09/22/2014
• Issued: 06/07/2016
• Est. Priority Date: 08/01/2013
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cook, Kelly L., Rodriguez, Adrian X., Skeen, Michael M., Woods, Eric, Wuu, Menlo
• Primary Examiner(s): Hill, Stanley K
• Assistant Examiner(s): Smith, Paulinho E
• Application Number:
  US14/492,463
• Publication Number:
  US 20150058329A1
• Time in Patent Office: 624 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

Mechanisms for clarifying an input question are provided. A question is received for generation of an answer. A set of candidate answers is generated based on an analysis of a corpus of information. Each candidate answer has an evidence passage supporting the candidate answer. Based on the set of candidate answers, a determination is made as to whether clarification of the question is required. In response to a determination that clarification of the question is required, a request is sent for user input to clarify the question. User input is received from the computing device in response to the request and at least one candidate answer in the set of candidate answers is selected as an answer for the question based on the user input.

Claims

1. A method, in a data processing system comprising a processor and a memory, for clarifying an input question, the method comprising:

receiving, in the data processing system from a computing device, the input question for generation of an answer to the input question;

generating, in the data processing system, a set of candidate answers for the input question based on an analysis of a corpus of information, wherein each candidate answer in the set of candidate answers corresponds to an evidence passage supporting the candidate answer as answering the input question;

determining, in the data processing system, based on the set of candidate answers, whether clarification of the input question is required;

identifying, in response to a determination that clarification of the input question is required, a differentiating factor in evidence passages of at least two candidate answers in the set of candidate answers;

sending, by the data processing system, in response to a determination that clarification of the input question is required, a request for user input to clarify the input question, wherein the request for user input is generated based on the identified differentiating factor;

receiving, in the data processing system, user input from the computing device in response to the request; and

selecting, by the data processing system, at least one candidate answer in the set of candidate answers as an answer for the input question based on the user input, wherein identifying a differentiating factor in evidence passages of at least two candidate answers in the set of candidate answers comprises;

identifying a plurality of differentiating factors between evidence passages of the at least two candidate answers; and

selecting a subset of differentiating factors from the plurality of differentiating factors based on an evaluation of which differentiating factors in the plurality of differentiating factors clarify the input question.

2. The method of claim 1, wherein the request for user input comprises a clarification question directed to the differentiating factor and a plurality of user selectable potential answers to the clarification question, each answer corresponding to a portion of a corresponding one of the evidence passages, of the at least two candidate answers, directed to the differentiating factor.

3. The method of claim 1, wherein the request for user input comprises a clarification question that comprises a potential answer corresponding to the differentiating factor in the content of the clarification question and user selectable potential answers in the affirmative and negative for answering the clarification question.

4. The method of claim 1, wherein the request for user input comprises a clarification question that is directed to the differentiating factor and a free-form text entry field into which a user may input a textual answer to the clarification question.

6. The method of claim 1, wherein selecting at least one candidate answer in the set of candidate answers as an answer for the input question based on the user input comprises:

updating the set of candidate answers based on the user input; and

selecting the at least one candidate answer from the updated set of candidate answers.

7. The method of claim 6, wherein updating the set of candidate answers comprises modifying confidence scores associated with one or more of the candidate answers in the set of candidate answers based on the user input, wherein confidence scores for candidate answers having evidence passages corresponding to the user input are increased and candidate answers having evidence passages not corresponding to the user input are decreased.

8. The method of claim 6, wherein updating the set of candidate answers comprises removing candidate answers, from the set of candidate answers, that have evidence passages that do not correspond to the user input.

9. The method of claim 6, wherein selecting the at least one candidate answer from the updated set of candidate answers comprises:

performing synthesis stage, merging and ranking stage, and final answer selecting stage operations of a question and answer (QA) system pipeline on the updated set of candidate answers.

19. The method of claim 1, wherein the differentiating factor is a factor that differentiates a context of one candidate answer from another candidate answer in the at least two candidate answers.

20. The method of claim 1, further comprising calculating a frequency of occurrence of each of the differentiating factors in the plurality of differentiating factors in the evidence passages of the at least two candidate answers, wherein selecting the subset of differentiating factors comprises selecting the subset of differentiating factors based on the frequencies of occurrence associated with each of the differentiating factors.

21. The method of claim 20, wherein selecting the subset of differentiating factors based on the frequencies of occurrence associated with each of the differentiating factors comprises selecting differentiating factors from the plurality of differentiating factors, for inclusion in the subset of differentiating factors, that have associated frequencies of occurrence equal to or above a threshold value.

5. A method, in a data processing system comprising a processor and a memory, for clarifying an input question, the method comprising:

receiving, in the data processing system from a computing device, the input question for generation of an answer to the input question;

generating, in the data processing system, a set of candidate answers for the input question based on an analysis of a corpus of information, wherein each candidate answer in the set of candidate answers corresponds to an evidence passage supporting the candidate answer as answering the input question;

determining, in the data processing system, based on the set of candidate answers, whether clarification of the input question is required;

sending, by the data processing system, in response to a determination that clarification of the input question is required, a request for user input to clarify the input question;

receiving, in the data processing system, user input from the computing device in response to the request; and

selecting, by the data processing system, at least one candidate answer in the set of candidate answers as an answer for the input question based on the user input, wherein determining, based on the set of candidate answers, whether clarification of the input question is required comprises;

determining whether the set of candidate answers comprises more than one candidate answer with a corresponding confidence score equal to or higher than a predetermined threshold confidence score; and

determining that clarification of the input question is required in response to determining that the set of candidate answers comprises more than one candidate answer with a corresponding confidence score equal to or higher than the predetermined threshold confidence score.

10. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

receive an input question for generation of an answer to the input question;

generate a set of candidate answers for the input question based on an analysis of a corpus of information, wherein each candidate answer in the set of candidate answers corresponds to an evidence passage supporting the candidate answer as answering the input question;

determine, based on the set of candidate answers, whether clarification of the input question is required;

identify, in response to a determination that clarification of the input question is required, a differentiating factor in evidence passages of at least two candidate answers in the set of candidate answers;

send, in response to a determination that clarification of the input question is required, a request for user input to clarify the input question, wherein the request for user input is generated based on the identified differentiating factor;

receive user input from the computing device in response to the request; and

select at least one candidate answer in the set of candidate answers as an answer for the input question based on the user input wherein identifying a differentiating factor in evidence passages of at least two candidate answers in the set of candidate answers comprises;

identifying a plurality of differentiating factors between evidence passages of the at least two candidate answers; and

selecting a subset of differentiating factors from the plurality of differentiating factors based on an evaluation of which differentiating factors in the plurality of differentiating factors clarify the input question.

11. The apparatus of claim 10, wherein the request for user input comprises a clarification question directed to the differentiating factor and a plurality of user selectable potential answers to the clarification question, each answer corresponding to a portion of a corresponding one of the evidence passages, of the at least two candidate answers, directed to the differentiating factor.

12. The apparatus of claim 10, wherein the request for user input comprises a clarification question that comprises a potential answer corresponding to the differentiating factor in the content of the clarification question and user selectable potential answers in the affirmative and negative for answering the clarification question.

13. The apparatus of claim 10, wherein the request for user input comprises a clarification question that is directed to the differentiating factor and a free-form text entry field into which a user may input a textual answer to the clarification question.

15. The apparatus of claim 10, wherein selecting at least one candidate answer in the set of candidate answers as an answer for the input question based on the user input comprises:

updating the set of candidate answers based on the user input; and

selecting the at least one candidate answer from the updated set of candidate answers.

16. The apparatus of claim 15, wherein updating the set of candidate answers comprises modifying confidence scores associated with one or more of the candidate answers in the set of candidate answers based on the user input, wherein confidence scores for candidate answers having evidence passages corresponding to the user input are increased and candidate answers having evidence passages not corresponding to the user input are decreased.

17. The apparatus of claim 15, wherein updating the set of candidate answers comprises removing candidate answers, from the set of candidate answers, that have evidence passages that do not correspond to the user input.

18. The apparatus of claim 15, wherein selecting the at least one candidate answer from the updated set of candidate answers comprises:

performing synthesis stage, merging and ranking stage, and final answer selecting stage operations of a question and answer (QA) system pipeline on the updated set of candidate answers.

14. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

receive an input question for generation of an answer to the input question;

generate a set of candidate answers for the input question based on an analysis of a corpus of information, wherein each candidate answer in the set of candidate answers corresponds to an evidence passage supporting the candidate answer as answering the input question;

determine, based on the set of candidate answers, whether clarification of the input question is required;

send, in response to a determination that clarification of the input question is required, a request for user input to clarify the input question;

receive user input from the computing device in response to the request; and

select at least one candidate answer in the set of candidate answers as an answer for the input question based on the user input, wherein determining, based on the set of candidate answers, whether clarification of the input question is required comprises;

determining whether the set of candidate answers comprises more than one candidate answer with a corresponding confidence score equal to or higher than a predetermined threshold confidence score; and

determining that clarification of the input question is required in response to determining that the set of candidate answers comprises more than one candidate answer with a corresponding confidence score equal to or higher than the predetermined threshold confidence score.