Clarification of submitted questions in a question and answer system

• US 9,342,608 B2
• Filed: 08/01/2013
• Issued: 05/17/2016
• Est. Priority Date: 08/01/2013
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cook, Kelly L., Rodriguez, Adrian X., Skeen, Michael M., Woods, Eric, Wuu, Menlo
• Primary Examiner(s): Hill, Stanley K
• Assistant Examiner(s): Smith, Paulinho E
• Application Number:
  US13/957,128
• Publication Number:
  US 20150039536A1
• Time in Patent Office: 1,020 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

Mechanisms for clarifying an input question are provided. A question is received for generation of an answer. A set of candidate answers is generated based on an analysis of a corpus of information. Each candidate answer has an evidence passage supporting the candidate answer. Based on the set of candidate answers, a determination is made as to whether clarification of the question is required. In response to a determination that clarification of the question is required, a request is sent for user input to clarify the question. User input is received from the computing device in response to the request and at least one candidate answer in the set of candidate answers is selected as an answer for the question based on the user input.

Claims

1. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

receive an input question for generation of an answer to the input question;

generate a set of candidate answers for the input question based on an analysis of a corpus of information, wherein each candidate answer in the set of candidate answers corresponds to an evidence passage supporting the candidate answer as answering the input question;

determine, based on the set of candidate answers, whether clarification of the input question is required;

identify, in response to a determination that clarification of the input question is required, a differentiating factor in evidence passages of at least two candidate answers in the set of candidate answers;

send, by the data processing system, in response to a determination that clarification of the input question is required, a request for user input to clarify the input question, wherein the request for user input is generated based on the identified differentiating factor;

receive user input from the computing device in response to the request; and

select at least one candidate answer in the set of candidate answers as an answer for the input question based on the user input, wherein identifying a differentiating factor in evidence passages of at least two candidate answers in the set of candidate answers comprises;

identifying a plurality of differentiating factors between evidence passages of the at least two candidate answers; and

selecting a subset of differentiating factors from the plurality of differentiating factors based on an evaluation of which differentiating factors in the plurality of differentiating factors clarify the input question.

2. The computer program product of claim 1, wherein the request for user input comprises a clarification question directed to the differentiating factor and a plurality of user selectable potential answers to the clarification question, each answer corresponding to a portion of a corresponding one of the evidence passages, of the at least two candidate answers, directed to the differentiating factor.

3. The computer program product of claim 1, wherein the request for user input comprises a clarification question that comprises a potential answer corresponding to the differentiating factor in the content of the clarification question and user selectable potential answers in the affirmative and negative for answering the clarification question.

4. The computer program product of claim 1, wherein the request for user input comprises a clarification question that is directed to the differentiating factor and a free-form text entry field into which a user may input a textual answer to the clarification question.

6. The computer program product of claim 1, wherein the computer readable program further causes the computing device to select at least one candidate answer in the set of candidate answers as an answer for the input question based on the user input at least by:

updating the set of candidate answers based on the user input; and

selecting the at least one candidate answer from the updated set of candidate answers.

7. The computer program product of claim 6, wherein the computer readable program further causes the computing device to update the set of candidate answers at least by modifying confidence scores associated with one or more of the candidate answers in the set of candidate answers based on the user input, wherein confidence scores for candidate answers having evidence passages corresponding to the user input are increased and candidate answers having evidence passages not corresponding to the user input are decreased.

8. The computer program product of claim 6, wherein the computer readable program further causes the computing device to update the set of candidate answers at least by removing candidate answers, from the set of candidate answers, that have evidence passages that do not correspond to the user input.

16. The computer program product of claim 1, wherein the differentiating factor is a factor that differentiates a context of one candidate answer from another candidate answer in the at least two candidate answers.

17. The computer program product of claim 1, further comprising calculating a frequency of occurrence of each of the differentiating factors in the plurality of differentiating factors in the evidence passages of the at least two candidate answers, wherein selecting the subset of differentiating factors comprises selecting the subset of differentiating factors based on the frequencies of occurrence associated with each of the differentiating factors.

18. The computer program product of claim 17, wherein selecting the subset of differentiating factors based on the frequencies of occurrence associated with each of the differentiating factors comprises selecting differentiating factors from the plurality of differentiating factors, for inclusion in the subset of differentiating factors, that have associated frequencies of occurrence equal to or above a threshold value.

5. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

receive an input question for generation of an answer to the input question;

generate a set of candidate answers for the input question based on an analysis of a corpus of information, wherein each candidate answer in the set of candidate answers corresponds to an evidence passage supporting the candidate answer as answering the input question;

determine, based on the set of candidate answers, whether clarification of the input question is required at least by;

determining whether the set of candidate answers comprises more than one candidate answer with a corresponding confidence score equal to or higher than a predetermined threshold confidence score; and

determining that clarification of the input question is required in response to determining that the set of candidate answers comprises more than one candidate answer with a corresponding confidence score equal to or higher than the predetermined threshold confidence score;

send, in response to a determination that clarification of the input question is required, a request for user input to clarify the input question;

receive user input from the computing device in response to the request; and

select at least one candidate answer in the set of candidate answers as an answer for the input question based on the user input.

9. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

receive an input question for generation of an answer to the input question;

generate a set of candidate answers for the input question based on an analysis of a corpus of information, wherein each candidate answer in the set of candidate answers corresponds to an evidence passage supporting the candidate answer as answering the input question;

determine, based on the set of candidate answers, whether clarification of the input question is required;

identify, in response to a determination that clarification of the input question is required, a differentiating factor in evidence passages of at least two candidate answers in the set of candidate answers;

send, in response to a determination that clarification of the input question is required, a request for user input to clarify the input question, wherein the request for user input is generated based on the identified differentiating factor;

receive user input from the computing device in response to the request; and

select at least one candidate answer in the set of candidate answers as an answer for the input question based on the user input.

10. The apparatus of claim 9, wherein the request for user input comprises a clarification question directed to the differentiating factor and a plurality of user selectable potential answers to the clarification question, each answer corresponding to a portion of a corresponding one of the evidence passages, of the at least two candidate answers, directed to the differentiating factor.

11. The apparatus of claim 9, wherein the request for user input comprises a clarification question that comprises a potential answer corresponding to the differentiating factor in the content of the clarification question and user selectable potential answers in the affirmative and negative for answering the clarification question.

12. The apparatus of claim 9, wherein the request for user input comprises a clarification question that is directed to the differentiating factor and a free-form text entry field into which a user may input a textual answer to the clarification question.

13. The apparatus of claim 9, wherein the instructions further cause the processor to select at least one candidate answer in the set of candidate answers as an answer for the input question based on the user input at least by:

updating the set of candidate answers based on the user input; and

selecting the at least one candidate answer from the updated set of candidate answers.

14. The apparatus of claim 13, wherein the instructions further cause the processor to update the set of candidate answers at least by modifying confidence scores associated with one or more of the candidate answers in the set of candidate answers based on the user input, wherein confidence scores for candidate answers having evidence passages corresponding to the user input are increased and candidate answers having evidence passages not corresponding to the user input are decreased.

15. The apparatus of claim 13, wherein the instructions further cause the processor to update the set of candidate answers at least by removing candidate answers, from the set of candidate answers, that have evidence passages that do not correspond to the user input.