Clarification of submitted questions in a question and answer system

• US 9,721,205 B2
• Filed: 05/02/2016
• Issued: 08/01/2017
• Est. Priority Date: 08/01/2013
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cook, Kelly L., Rodriguez, Adrian X., Skeen, Michael M., Woods, Eric, Wuu, Menlo
• Primary Examiner(s): Smith, Paulinho E
• Application Number:
  US15/143,771
• Publication Number:
  US 20160314393A1
• Time in Patent Office: 456 Days
• Field of Search: None
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

Mechanisms for clarifying an input question are provided. A question is received for generation of an answer. A set of candidate answers is generated based on an analysis of a corpus of information. Each candidate answer has an evidence passage supporting the candidate answer. Based on the set of candidate answers, a determination is made as to whether clarification of the question is required. In response to a determination that clarification of the question is required, a request is sent for user input to clarify the question. User input is received from the computing device in response to the request and at least one candidate answer in the set of candidate answers is selected as an answer for the question based on the user input.

Claims

1. A method, in a data processing system comprising a processor and a memory, for clarifying an input question, the method comprising:

generating, in the data processing system, a set of candidate answers for an input question, wherein each candidate answer in the set of candidate answers corresponds to an evidence passage supporting the candidate answer as answering the input question;

determining, in the data processing system, based on the set of candidate answers, whether clarification of the input question is required; and

in response to a determination that clarification of the input question is required;

identifying, by the data processing system, a differentiating factor in evidence passage of at least two candidate answers in the set of candidate answers;

outputting, by the data processing system, a request for user input to clarify the input question, wherein the request for user input is generated based on the identified differentiating factor; and

selecting, by the data processing system, at least one candidate answer in the set of candidate answers as an answer for the input question based on a user input in response to the request, wherein determining, based on the set of candidate answers, whether clarification of the input question is required comprises;

determining whether the set of candidate answers comprises more than one candidate answer with a corresponding confidence score equal to or higher than a predetermined threshold confidence score; and

determining that clarification of the input question is required in response to determining that the set of candidate answers comprises more than one candidate answer with a corresponding confidence score equal to or higher than the predetermined threshold confidence score.

2. The method of claim 1, wherein the request for user input comprises a clarification question directed to the differentiating factor and a plurality of user selectable potential answers to the clarification question, each answer corresponding to a portion of a corresponding one of the evidence passages, of the at least two candidate answers, directed to the differentiating factor.

3. The method of claim 1, wherein the request for user input comprises a clarification question that comprises a potential answer corresponding to the differentiating factor in the content of the clarification question and user selectable potential answers in the affirmative and negative for answering the clarification question.

4. The method of claim 1, wherein the request for user input comprises a clarification question that is directed to the differentiating factor and a free-form text entry field into which a user may input a textual answer to the clarification question.

5. The method of claim 1, wherein selecting at least one candidate answer in the set of candidate answers as an answer for the input question comprises:

updating the set of candidate answers based on the user input; and

selecting the at least one candidate answer from the updated set of candidate answers.

6. The method of claim 5, wherein updating the set of candidate answers comprises modifying confidence scores associated with one or more of the candidate answers in the set of candidate answers based on the user input, wherein confidence scores for candidate answers having evidence passages corresponding to the user input are increased and candidate answers having evidence passages not corresponding to the user input are decreased.

7. The method of claim 5, wherein updating the set of candidate answers comprises removing candidate answers, from the set of candidate answers, that have evidence passages that do not correspond to the user input.

8. The method of claim 5, wherein selecting the at least one candidate answer from the updated set of candidate answers comprises:

performing synthesis stage, merging and ranking stage, and final answer selecting stage operations of a question and answer (QA) system pipeline on the updated set of candidate answers.

9. The method of claim 1, wherein the request comprises a clarifying question posed to a user, wherein the clarifying question is generated based on the differentiating factor and is constructed such that an answer to the clarifying question indicates a correctness of one of the at least two candidate answers based on their associated evidence passages.

10. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

generate a set of candidate answers for an input question, wherein each candidate answer in the set of candidate answers corresponds to an evidence passage supporting the candidate answer as answering the input question;

determine, based on the set of candidate answers, whether clarification of the input question is required; and

in response to a determination that clarification of the input question is required;

identify a differentiation factor in evidence passages of at least two candidate answers in the set of candidate answers;

output a request for user input to clarify the input question, wherein the request for user input is generated based on the identified differentiating factor; and

select at least one candidate answer in the set of candidate answers as an answer for the input question based on a user response to the request, wherein the computer readable program further causes the computing device to determine, based on the set of candidate answers, whether clarification of the input question is required at least by;

determining whether the set of candidate answers comprises more than one candidate answer with a corresponding confidence score equal to or higher than a predetermined threshold confidence score; and

determining that clarification of the input question is required in response to determining that the set of candidate answers comprises more than one candidate answer with a corresponding confidence score equal to or higher than the predetermined threshold confidence score.

11. The computer program product of claim 10, wherein the request for user input comprises a clarification question directed to the differentiating factor and a plurality of user selectable potential answers to the clarification question, each answer corresponding to a portion of a corresponding one of the evidence passages, of the at least two candidate answers, directed to the differentiating factor.

12. The computer program product of claim 10, wherein the request for user input comprises a clarification question that comprises a potential answer corresponding to the differentiating factor in the content of the clarification question and user selectable potential answers in the affirmative and negative for answering the clarification question.

13. The computer program product of claim 10, wherein the request for user input comprises a clarification question that is directed to the differentiating factor and a free-form text entry field into which a user may input a textual answer to the clarification question.

14. The computer program product of claim 10, whefein the computer readable program further causes the computing device to select at least one candidate answer in the set of candidate answers as an answer for the input question at least by:

updating the set of candidate answers based on the user input; and

selecting the at least one candidate answer from the updated set of candidate answers.

15. The computer program product of claim 14, wherein the computer readable program further causes the computing device to update the set of candidate answers at least by modifying confidence scores associated with one or more of the candidate answers in the set of candidate answers based on the user input, wherein confidence scores for candidate answers having evidence passages corresponding to the user input are increased and candidate answers having evidence passages not corresponding to the user input are decreased.

16. The computer program product of claim 14, wherein the computer readable program further causes the computing device to update the set of candidate answers at least by removing candidate answers, from the set of candidate answers, that have evidence passages that do not correspond to the user input.

17. The computer program product of claim 10, wherein the request comprises a clarifying question posed to a user, wherein the clarifying question is generated based on the differentiating factor and is constructed such that an answer to the clarifying question indicates a correctness of one of the at least two candidate answers based on their associated evidence passages.

18. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

generate a set of candidate answers for an input question, wherein each candidate answer in the set of candidate answers corresponds to an evidence passage supporting the candidate answer as answering the input question;

determine, based on the set of candidate answers, whether clarification of the input question is required; and

in response to a determination that clarification of the input question is required;

identify a differentiation factor in evidence passages of at least two candidate answers in the set of candidate answers;

output a request for user input to clarify the input question, wherein the request for user input is generated based on the identified differentiating factor; and

select at least one candidate answer in the set of candidate answers as an answer for the input question based on a user response to the request, wherein the instructions further cause the processor to determine, based on the set of candidate answers, whether clarification of the input question is required at least by;

determining whether the set of candidate answers comprises more than one candidate answer with a corresponding confidence score equal to or higher than a predetermined threshold confidence score; and

determining that clarification of the input Question is required in response to determining that the set of candidate answers comprises more than one candidate answer with a corresponding confidence score equal to or hi er than the predetermined threshold confidence score.