Clarification of Submitted Questions in a Question and Answer System

• US 20150058329A1
• Filed: 09/22/2014
• Published: 02/26/2015
• Est. Priority Date: 08/01/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cook, Kelly L., Rodriguez, Adrian X., Skeen, Michael M., Woods, Eric, Wuu, Menlo
• Granted Patent: US 9,361,386 B2
• Time in Patent Office: Days
• US Class Current: 707/723
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

Mechanisms for clarifying an input question are provided. A question is received for generation of an answer. A set of candidate answers is generated based on an analysis of a corpus of information. Each candidate answer has an evidence passage supporting the candidate answer. Based on the set of candidate answers, a determination is made as to whether clarification of the question is required. In response to a determination that clarification of the question is required, a request is sent for user input to clarify the question. User input is received from the computing device in response to the request and at least one candidate answer in the set of candidate answers is selected as an answer for the question based on the user input.

Claims

1. A method, in a data processing system comprising a processor and a memory, for clarifying an input question, the method comprising:

receiving, in the data processing system from a computing device, the input question for generation of an answer to the input question;

generating, in the data processing system, a set of candidate answers for the input question based on an analysis of a corpus of information, wherein each candidate answer in the set of candidate answers corresponds to an evidence passage supporting the candidate answer as answering the input question;

determining, in the data processing system, based on the set of candidate answers, whether clarification of the input question is required;

sending, by the data processing system, in response to a determination that clarification of the input question is required, a request for user input to clarify the input question;

receiving, in the data processing system, user input from the computing device in response to the request; and

selecting, by the data processing system, at least one candidate answer in the set of candidate answers as an answer for the input question based on the user input.

2. The method of claim 1, further comprising:

identifying, in response to a determination that clarification of the input question is required, a differentiating factor in evidence passages of at least two candidate answers in the set of candidate answers, wherein the request for user input is generated based on the identified differentiating factor.

3. The method of claim 2, wherein the request for user input comprises a clarification question directed to the differentiating factor and a plurality of user selectable potential answers to the clarification question, each answer corresponding to a portion of a corresponding one of the evidence passages, of the at least two candidate answers, directed to the differentiating factor.

4. The method of claim 2, wherein the request for user input comprises a clarification question that comprises a potential answer corresponding to the differentiating factor in the content of the clarification question and user selectable potential answers in the affirmative and negative for answering the clarification question.

5. The method of claim 2, wherein the request for user input comprises a clarification question that is directed to the differentiating factor and a free-form text entry field into which a user may input a textual answer to the clarification question.

6. The method of claim 1, wherein determining, based on the set of candidate answers, whether clarification of the input question is required comprises:

determining whether the set of candidate answers comprises more than one candidate answer with a corresponding confidence score equal to or higher than a predetermined threshold confidence score; and

determining that clarification of the input question is required in response to determining that the set of candidate answers comprises more than one candidate answer with a corresponding confidence score equal to or higher than the predetermined threshold confidence score.

7. The method of claim 1, wherein selecting at least one candidate answer in the set of candidate answers as an answer for the input question based on the user input comprises:

updating the set of candidate answers based on the user input; and

selecting the at least one candidate answer from the updated set of candidate answers.

8. The method of claim 7, wherein updating the set of candidate answers comprises modifying confidence scores associated with one or more of the candidate answers in the set of candidate answers based on the user input, wherein confidence scores for candidate answers having evidence passages corresponding to the user input are increased and candidate answers having evidence passages not corresponding to the user input are decreased.

9. The method of claim 7, wherein updating the set of candidate answers comprises removing candidate answers, from the set of candidate answers, that have evidence passages that do not correspond to the user input.

10. The method of claim 7, wherein selecting the at least one candidate answer from the updated set of candidate answers comprises:

performing synthesis stage, merging and ranking stage, and final answer selecting stage operations of a question and answer (QA) system pipeline on the updated set of candidate answers.

11. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

receive an input question for generation of an answer to the input question;

generate a set of candidate answers for the input question based on an analysis of a corpus of information, wherein each candidate answer in the set of candidate answers corresponds to an evidence passage supporting the candidate answer as answering the input question;

determine, based on the set of candidate answers, whether clarification of the input question is required;

send, in response to a determination that clarification of the input question is required, a request for user input to clarify the input question;

receive user input from the computing device in response to the request; and

select at least one candidate answer in the set of candidate answers as an answer for the input question based on the user input.

12. The computer program product of claim 11, wherein the computer readable program further causes the computing device to:

identify, in response to a determination that clarification of the input question is required, a differentiating factor in evidence passages of at least two candidate answers in the set of candidate answers, wherein the request for user input is generated based on the identified differentiating factor.

13. The computer program product of claim 12, wherein the request for user input comprises a clarification question directed to the differentiating factor and a plurality of user selectable potential answers to the clarification question, each answer corresponding to a portion of a corresponding one of the evidence passages, of the at least two candidate answers, directed to the differentiating factor.

14. The computer program product of claim 12, wherein the request for user input comprises a clarification question that comprises a potential answer corresponding to the differentiating factor in the content of the clarification question and user selectable potential answers in the affirmative and negative for answering the clarification question.

15. The computer program product of claim 12, wherein the request for user input comprises a clarification question that is directed to the differentiating factor and a free-form text entry field into which a user may input a textual answer to the clarification question.

16. The computer program product of claim 11, wherein the computer readable program further causes the computing device to determine, based on the set of candidate answers, whether clarification of the input question is required at least by:

determining whether the set of candidate answers comprises more than one candidate answer with a corresponding confidence score equal to or higher than a predetermined threshold confidence score; and

determining that clarification of the input question is required in response to determining that the set of candidate answers comprises more than one candidate answer with a corresponding confidence score equal to or higher than the predetermined threshold confidence score.

17. The computer program product of claim 11, wherein the computer readable program further causes the computing device to select at least one candidate answer in the set of candidate answers as an answer for the input question based on the user input at least by:

updating the set of candidate answers based on the user input; and

selecting the at least one candidate answer from the updated set of candidate answers.

18. The computer program product of claim 17, wherein the computer readable program further causes the computing device to update the set of candidate answers at least by modifying confidence scores associated with one or more of the candidate answers in the set of candidate answers based on the user input, wherein confidence scores for candidate answers having evidence passages corresponding to the user input are increased and candidate answers having evidence passages not corresponding to the user input are decreased.

19. The computer program product of claim 17, wherein the computer readable program further causes the computing device to update the set of candidate answers at least by removing candidate answers, from the set of candidate answers, that have evidence passages that do not correspond to the user input.

20. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

receive an input question for generation of an answer to the input question;

generate a set of candidate answers for the input question based on an analysis of a corpus of information, wherein each candidate answer in the set of candidate answers corresponds to an evidence passage supporting the candidate answer as answering the input question;

determine, based on the set of candidate answers, whether clarification of the input question is required;

send, in response to a determination that clarification of the input question is required, a request for user input to clarify the input question;

receive user input from the computing device in response to the request; and

select at least one candidate answer in the set of candidate answers as an answer for the input question based on the user input.