HANDLING INFORMATION SOURCE INGESTION IN A QUESTION ANSWERING SYSTEM

• US 20160048514A1
• Filed: 08/13/2014
• Published: 02/18/2016
• Est. Priority Date: 08/13/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Allen, Corville O., Jamrog, Daniel M., Witherspoon, Kristin A.
• Granted Patent: US 9,703,840 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/954 Navigation, e.g. using categorised browsing

Abstract

A question-answering (QA) system first receives input questions. Each question is then assigned to a first question category of a plurality of question categories. The QA system then identifies a set of candidate answers to each question using a core information source. A set of confidence scores, including a confidence score for each candidate answer, is then calculated. The QA system then determines that the first set of confidence scores fails to satisfy confidence criteria. In response to this determination, an updated information source is ingested.

Claims

1. A computer-implemented method for handling a plurality of input questions, the method comprising:

receiving, using a computer, the plurality of input questions;

assigning, using the computer, each question of the plurality of input questions to a first question category of a plurality of question categories;

identifying, using the computer, a set of candidate answers to each input question of the plurality of input questions using a core information source;

calculating, using the computer, a first set of confidence scores, the first set of confidence scores including a confidence score for each candidate answer of each set of candidate answers; and

in response to determining that the first set of confidence scores fails to satisfy confidence criteria, ingesting an updated information source.

2. The method of claim 1, further comprising:

prior to the ingesting the updated information source, determining, using the computer, that the updated information source has an association with the first question category; and

selecting the updated information source for ingestion based on at least the association between the updated information source and the first question category.

3. The method of claim 1, further comprising:

receiving, using the computer, a second plurality of input questions;

assigning, using the computer, the second plurality of input questions to a second question category of the plurality of question categories;

identifying, using the computer, a set of candidate answers to each input question of the second plurality of input questions using the core information source;

calculating, using the computer, a second set of confidence scores, the second set of confidence scores including a confidence score for each candidate answer of each set of candidate answers to each input question of the second plurality of input questions; and

determining, using the computer, that the second set of confidence scores satisfies the confidence criteria.

4. The method of claim 1, wherein each question of the plurality of input questions are assigned to the first question category based on at least determining, using the computer, a type and a class for each question and analyzing, using the computer, the type and the class for each question.

5. The method of claim 1, further comprising:

receiving, using the computer, an audit trigger;

in response to the receiving, selecting, using the computer, the first question category for an audit;

selecting, using the computer, an interval for the audit; and

during the audit, compiling and analyzing, using the computer, the first set of confidence scores.

6. The method of claim 1, further comprising:

prior to the receiving the plurality of input questions, identifying, using the computer, a first set of training candidate answers to each training question of a plurality of training questions using the original information source; and

subsequent to the ingesting the updated information source, identifying, using the computer, a second set of training candidate answers to each training question of the plurality of training questions using the updated information source.

7. The method of claim 6, further comprising:

receiving, using the computer, feedback from a user, the feedback including an indication that none of a set of candidate answers to a first input question of the plurality of input questions is accurate, and the feedback further including a correct answer to the first input question; and

in response to the receiving the feedback, identifying, using the computer, the correct answer to the first input question using the updated information source.

8. The method of claim 6, further comprising:

subsequent to the identifying the second set of training candidate answers, identifying, using the computer, a set of candidate answers to each question of a second plurality of input questions using the updated information source.

9. The method of claim 8, wherein, prior to the identifying the set of candidate answers to each question of the second plurality of input questions, the method further comprises:

comparing, using the computer, the first set of training answers and the second set of training answers;

based on the comparison, determining, using the computer, that differences between the first set of training answers and the second set of training answers justify adding the updated information source to the core information source; and

adding, using the computer, the updated information source to the core information source.

10. The method of claim 8, wherein, prior to the identifying the set of candidate answers to each question of the second plurality of input questions, the method further comprises:

calculating, using the computer, a set of training confidence scores, the set of training confidence scores including a confidence score for each candidate answer of the second set of training candidate answers to each training question of the plurality of training questions; and

determining, using the computer, that the set of training confidence scores satisfies training confidence criteria.

11. A system for handling a plurality of input questions, the system comprising:

a computer configured to;

receive the plurality of input questions;

assign each question of the plurality of input questions to a first question category of a plurality of question categories;

identify a set of candidate answers to each input question of the plurality of input questions using a core information source;

calculate a first set of confidence scores, the first set of confidence scores including a confidence score for each candidate answer of each set of candidate answers; and

in response to determining that the first set of confidence scores fails to satisfy confidence criteria, ingest an updated information source.

12. The system of claim 11, wherein the computer is further configured to:

prior to the ingesting the updated information source, determine that the updated information source has an association with the first question category; and

select the updated information source for ingestion based on at least the association between the updated information source and the first question category.

13. The system of claim 11, wherein the computer is further configured to:

receive an audit trigger;

in response to the receiving, select the first question category for an audit;

select an interval for the audit; and

during the audit, compile and analyze the first set of confidence scores.

14. The system of claim 11, wherein the computer is further configured to:

prior to the receiving the plurality of input questions, identify a first set of training candidate answers to each training question of a plurality of training questions using the original information source; and

subsequent to the ingesting the updated information source, identify a second set of training candidate answers to each training question of the plurality of training questions using the updated information source.

15. The system of claim 14, wherein the computer is further configured to:

receive feedback from a user, the feedback including an indication that none of a set of candidate answers to a first input question of the plurality of input questions is accurate, and the feedback further including a correct answer to the first input question; and

in response to the receiving the feedback, identify the correct answer to the first input question using the updated information source.

16. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

receiving a plurality of input questions;

assigning each question of the plurality of input questions to a first question category of a plurality of question categories;

identifying a set of candidate answers to each input question of the plurality of input questions using a core information source;

calculating a first set of confidence scores, the first set of confidence scores including a confidence score for each candidate answer of each set of candidate answers;

in response to determining that the first set of confidence scores fails to satisfy confidence criteria, ingesting an updated information source.

17. The computer program product of claim 16, wherein the program instructions are further executable by the computer to cause the computer to perform the method further comprising:

prior to the ingesting the updated information source, determining that the updated information source has an association with the first question category; and

selecting the updated information source for ingestion based on at least the association between the updated information source and the first question category.

18. The computer program product of claim 16, wherein the program instructions are further executable by the computer to cause the computer to perform the method further comprising:

receiving an audit trigger;

in response to the receiving, selecting the first question category for an audit;

selecting an interval for the audit; and

during the audit, compiling and analyzing the first set of confidence scores.

19. The computer program product of claim 16, wherein the program instructions are further executable by the computer to cause the computer to perform the method further comprising:

prior to the receiving the plurality of input questions, identifying a first set of training candidate answers to each training question of a plurality of training questions using the original information source; and

subsequent to the ingesting the updated information source, identifying a second set of training candidate answers to each training question of the plurality of training questions using the updated information source.

20. The computer program product of claim 19, wherein the program instructions are further executable by the computer to cause the computer to perform the method further comprising:

subsequent to the identifying the second set of training candidate answers, identifying a set of candidate answers to each question of a second plurality of input questions using the updated information source.