Results of Question and Answer Systems

• US 20140272909A1
• Filed: 03/15/2013
• Published: 09/18/2014
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Schwartz, David R., Taieb, David D., Vardy, Cale R., Isensee, Scott H., Johnson, Robert C. Jr.
• Granted Patent: US 9,280,908 B2
• Time in Patent Office: Days
• US Class Current: 434/362
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

Mechanisms are provided for modifying an operation of a question answering (QA) system. An input question is received and processed to generate at least one query to be applied to a corpus of information. The at least one query is applied to the corpus of information to generate candidate answers to the input question from which a final answer is selected for output. A training engine modifies, using a machine learning technique that compares the final answer to a known correct answer for the input question, at least one of logic or configuration parameters of the QA system for at least one of the processing of the input question to generate the at least one query, applying of the at least one query to the corpus of information to generate the candidate answers, or the selecting of the final answer from the candidate answers.

Claims

1. A method, in a data processing system comprising a processor and a memory, for modifying an operation of a question answering (QA) system, comprising:

receiving, by the data processing system, an input question;

processing, by the data processing system, the input question to generate at least one query to be applied to a corpus of information;

applying, by the data processing system, the at least one query to the corpus of information to generate candidate answers to the input question;

selecting, by the data processing system, a final answer from the candidate answers for output; and

modifying, by a training engine associated with the data processing system, using a machine learning technique that compares the final answer to a known correct answer known to be a correct answer for the input question, at least one of logic or configuration parameters of the QA system for at least one of the processing of the input question to generate the at least one query, applying of the at least one query to the corpus of information to generate the candidate answers, or the selecting of the final answer from the candidate answers.

2. The method of claim 1, wherein modifying, using a machine learning technique, at least one of logic or configuration parameters of the QA system comprises:

presenting the candidate answers via a user interface comprising elements for receiving user feedback ratings of the candidate answers;

receiving, via the user interface, user input specifying user feedback rating values for at least one of the candidate answers; and

modifying at least one of the logic or configuration parameters of the QA system based on the received user input specifying the user feedback rating values.

3. The method of claim 2, wherein the user feedback for a candidate answer identifies a user'"'"'s perceived usefulness or quality of the candidate answer.

4. The method of claim 2, wherein modifying at least one of the logic or configuration parameters of the QA system, comprises, for each candidate answer in the candidate answers:

determining a difference between the candidate answer and the known correct answer;

determining a user feedback rating value corresponding to the candidate answer; and

modifying logic or configuration parameters of the QA system so as to modify a computation of a confidence measure associated with the candidate answer based on both the determined difference and the user feedback rating value.

5. The method of claim 2, further comprising:

accumulating, for an answer source, user rating values for candidate answers generated based on content provided by the answer source to generate an accumulated user rating value for the answer source; and

modifying confidence measures associated with candidate answers generated based on content provided by the answer source based on the accumulated user rating value.

6. The method of claim 1, wherein modifying, using a machine learning technique, at least one of logic or configuration parameters of the QA system comprises:

generating one or more alternative versions of the input question;

processing the one or more alternative versions of the input question using the QA system; and

generating one or more candidate answers for each of the one or more alternative versions of the input question, wherein selecting a final answer from the one or more candidate answers comprises selecting a final answer from amongst all of the candidate answers generated for each of the input question and the one or more alternative versions of the input question.

7. The method of claim 6, wherein generating one or more alternative versions of the input question comprises at least one of reordering words in the input question, changing an inflection of a word in the input question, or replacing a word in the input question with a synonym.

8. The method of claim 6, wherein selecting a final answer from amongst all of the candidate answers generated for each of the input question and the one or more alternative versions of the input question comprises selecting a highest confidence measure question from amongst the input question and the one or more alternative versions of the input question.

9. The method of claim 6, further comprising:

inputting, to the training engine, the selected final answer from amongst all of the candidate answers generated for each of the input question and the one or more alternative versions of the input question; and

modifying logic or configuration parameters of the QA system based on the input of the selected final answer from amongst all of the candidate answers generated for each of the input question and the one or more alternative versions of the input question.

10. The method of claim 1, wherein modifying, using a machine learning technique, at least one of logic or configuration parameters of the QA system comprises:

logging the at least one query in a log data structure in association with an identifier of the input question and a source of the input question;

performing a search of the log data structure based on the at least one query to identify one or more matching entries in the log data structure having at least one matching query in the at least one query; and

outputting a notification of the one or more matching entries based on results of performing the search of the log data structure.

11. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

receive an input question;

process the input question to generate at least one query to be applied to a corpus of information;

apply the at least one query to the corpus of information to generate candidate answers to the input question;

select a final answer from the candidate answers for output; and

modify, by a training engine associated with the data processing system, using a machine learning technique that compares the final answer to a known correct answer known to be a correct answer for the input question, at least one of logic or configuration parameters of the QA system for at least one of the processing of the input question to generate the at least one query, applying of the at least one query to the corpus of information to generate the candidate answers, or the selecting of the final answer from the candidate answers.

12. The computer program product of claim 11, wherein modifying, using a machine learning technique, at least one of logic or configuration parameters of the QA system comprises:

presenting the candidate answers via a user interface comprising elements for receiving user feedback ratings of the candidate answers;

receiving, via the user interface, user input specifying user feedback rating values for at least one of the candidate answers; and

modifying at least one of the logic or configuration parameters of the QA system based on the received user input specifying the user feedback rating values.

13. The computer program product of claim 12, wherein the user feedback for a candidate answer identifies a user'"'"'s perceived usefulness or quality of the candidate answer.

14. The computer program product of claim 12, wherein modifying at least one of the logic or configuration parameters of the QA system, comprises, for each candidate answer in the candidate answers:

determining a difference between the candidate answer and the known correct answer;

determining a user feedback rating value corresponding to the candidate answer; and

modifying logic or configuration parameters of the QA system so as to modify a computation of a confidence measure associated with the candidate answer based on both the determined difference and the user feedback rating value.

15. The computer program product of claim 12, wherein the computer readable program further causes the computing device to:

accumulate, for an answer source, user rating values for candidate answers generated based on content provided by the answer source to generate an accumulated user rating value for the answer source; and

modify confidence measures associated with candidate answers generated based on content provided by the answer source based on the accumulated user rating value.

16. The computer program product of claim 11, wherein modifying, using a machine learning technique, at least one of logic or configuration parameters of the QA system comprises:

generating one or more alternative versions of the input question;

processing the one or more alternative versions of the input question using the QA system; and

generating one or more candidate answers for each of the one or more alternative versions of the input question, wherein selecting a final answer from the one or more candidate answers comprises selecting a final answer from amongst all of the candidate answers generated for each of the input question and the one or more alternative versions of the input question.

17. The computer program product of claim 16, wherein generating one or more alternative versions of the input question comprises at least one of reordering words in the input question, changing an inflection of a word in the input question, or replacing a word in the input question with a synonym.

18. The computer program product of claim 16, wherein selecting a final answer from amongst all of the candidate answers generated for each of the input question and the one or more alternative versions of the input question comprises selecting a highest confidence measure question from amongst the input question and the one or more alternative versions of the input question.

19. The computer program product of claim 16, wherein the computer readable program further causes the computing device to:

input, to the training engine, the selected final answer from amongst all of the candidate answers generated for each of the input question and the one or more alternative versions of the input question; and

modify logic or configuration parameters of the QA system based on the input of the selected final answer from amongst all of the candidate answers generated for each of the input question and the one or more alternative versions of the input question.

20. The computer program product of claim 11, wherein modifying, using a machine learning technique, at least one of logic or configuration parameters of the QA system comprises:

logging the at least one query in a log data structure in association with an identifier of the input question and a source of the input question;

performing a search of the log data structure based on the at least one query to identify one or more matching entries in the log data structure having at least one matching query in the at least one query; and

outputting a notification of the one or more matching entries based on results of performing the search of the log data structure.

21. An apparatus, comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

receive an input question;

process the input question to generate at least one query to be applied to a corpus of information;

apply the at least one query to the corpus of information to generate candidate answers to the input question;

select a final answer from the candidate answers for output; and

modify, by a training engine associated with the data processing system, using a machine learning technique that compares the final answer to a known correct answer known to be a correct answer for the input question, at least one of logic or configuration parameters of the QA system for at least one of the processing of the input question to generate the at least one query, applying of the at least one query to the corpus of information to generate the candidate answers, or the selecting of the final answer from the candidate answers.

22. The apparatus of claim 21, wherein modifying, using a machine learning technique, at least one of logic or configuration parameters of the QA system comprises:

presenting the candidate answers via a user interface comprising elements for receiving user feedback ratings of the candidate answers;

receiving, via the user interface, user input specifying user feedback rating values for at least one of the candidate answers; and

modifying at least one of the logic or configuration parameters of the QA system based on the received user input specifying the user feedback rating values.

23. The apparatus of claim 21, wherein modifying, using a machine learning technique, at least one of logic or configuration parameters of the QA system comprises:

generating one or more alternative versions of the input question;

processing the one or more alternative versions of the input question using the QA system; and

generating one or more candidate answers for each of the one or more alternative versions of the input question, wherein selecting a final answer from the one or more candidate answers comprises selecting a final answer from amongst all of the candidate answers generated for each of the input question and the one or more alternative versions of the input question.

24. The apparatus of claim 21, wherein modifying, using a machine learning technique, at least one of logic or configuration parameters of the QA system comprises:

logging the at least one query in a log data structure in association with an identifier of the input question and a source of the input question;

performing a search of the log data structure based on the at least one query to identify one or more matching entries in the log data structure having at least one matching query in the at least one query; and

outputting a notification of the one or more matching entries based on results of performing the search of the log data structure.