Concept noise reduction in deep question answering systems

• US 9,092,740 B2
• Filed: 11/08/2012
• Issued: 07/28/2015
• Est. Priority Date: 11/08/2012
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Petri, John E., Stevens, Richard J.
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): PELLETT, DANIEL T
• Application Number:
  US13/671,810
• Publication Number:
  US 20140129492A1
• Time in Patent Office: 992 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G09B 19/0053 Computers, e.g. programming

Abstract

Method, computer program product, and system to perform an operation for a deep question answering system. The operation begins by computing a concept score for a first concept in a first case received by the deep question answering system, the concept score being based on a machine learning concept model for the first concept. The operation then excludes the first concept from consideration when analyzing a candidate answer and an item of supporting evidence to generate a response to the first case upon determining that the concept score does not exceed a predefined concept minimum weight threshold. The operation then increases a weight applied to the first concept when analyzing the candidate answer and the item of supporting evidence to generate the response to the first case when the concept score exceeds a predefined maximum weight threshold.

Claims

1. A computer program product, comprising:

a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code comprising;

computer-readable program code configured to compute a concept score for a first concept in a first case, wherein the concept score is based on whether the first concept is present in at least one of a candidate answer and an item of supporting evidence;

computer-readable program code configured to adjust the concept score based on a coefficient for the first concept in a machine learning concept model for the first concept, wherein the coefficient reflects a relevance of the first concept in generating a correct response to the first case;

computer-readable program code configured to exclude the first concept from consideration when analyzing the candidate answer and the item of supporting evidence to generate a response to the first case upon determining at least one of;

(i) that the adjusted concept score does not exceed a predefined minimum weight threshold and (ii) that the coefficient does not exceed a predefined noise threshold; and

computer-readable program code configured to consider the first concept when analyzing the candidate answer and the item of supporting evidence to generate the response to the first case upon determining at least one of the concept score and the coefficient exceeds a predefined maximum weight threshold.

2. The computer program product of claim 1, further comprising:

computer-readable program code configured to compute a plurality of concept matching scores for a plurality of concepts in the candidate answer based on a respective coefficient for each concept specified by the machine learning concept model; and

computer-readable program code configured to aggregate the plurality of computed concept matching scores into a single relevancy score, wherein the single relevancy score is used to generate a response to the first case.

3. The computer program product of claim 1, further comprising computer-readable program code configured to train the deep question answering system to generate the machine learning concept model, wherein the computer-readable program code configured to train the deep question answering system comprises:

computer-readable program code configured to provide the deep question answering system with a sample case and a sample answer;

computer-readable program code configured to identify the first concept in the sample case;

computer-readable program code configured to compute the concept matching score for the first concept in the sample case based on the sample answer; and

computer-readable program code configured to include the concept matching score in the machine learning concept model.

4. The computer program product of claim 3, wherein producing the machine learning concept model further comprises identifying a type of question included in the sample case, wherein the type of question includes the first concept, wherein the concept matching score is based a degree of relevancy for the first concept and the type of question.

5. The computer program product of claim 1, further comprising:

computer-readable program code configured to generate, based on the first concept score, a response to the first case.

6. The computer program product of claim 1, wherein the concept score is further computed based on the presence of a relationship between the first concept in the first case and a second concept in the in the candidate answer.

7. The computer program product of claim 1, further comprising:

computer-readable program code configured to detect the first concept in the case.

8. The computer program product of claim 1, wherein the first concept is stored in an ontology comprising a plurality of concepts and concept relationships.

9. A system, comprising:

one or more computer processors; and

a memory containing a program, which, when executed by the one or more computer processors, performs an operation comprising;

computing a concept score for a first concept in a first case, wherein the concept score is based on whether the first concept is present in at least one of a candidate answer and an item of supporting evidence;

adjusting the concept score based on a coefficient for the first concept in a machine learning concept model for the first concept, wherein the coefficient reflects a relevance of the first concept in generating a correct response to the first case;

excluding the first concept from consideration when analyzing the candidate answer and the item of supporting evidence to generate a response to the first case upon determining at least one of;

(i) that the adjusted concept score does not exceed a predefined minimum weight threshold and (ii) that the coefficient does not exceed a predefined noise threshold; and

considering the first concept when analyzing the candidate answer and the item of supporting evidence to generate the response to the first case upon determining at least one of the concept score and the coefficient exceeds a predefined maximum weight threshold.

10. The system of claim 9, the operation further comprising:

computing a plurality of concept matching scores for a plurality of concepts in the candidate answer based on a respective weight factor specified by the machine learning concept model; and

aggregating the plurality of computed concept matching scores into a single relevancy score, wherein the single relevancy score is used to generate a response to the first case.

11. The system of claim 9, the operation further comprising training the deep question answering system to generate the machine learning concept model, wherein training the deep question answering system comprises:

providing the deep question answering system with a sample case and a sample answer;

identifying the first concept in the sample case;

computing the concept matching score for the first concept in the sample case based on the sample answer; and

including the concept matching score in the machine learning concept model.

12. The system of claim 11, wherein producing the machine learning concept model further comprises identifying a type of question included in the sample case, wherein the type of question includes the first concept, wherein the concept matching score is based a degree of relevancy for the first concept and the type of question.

13. The system of claim 9, the operation further comprising generating, based on the first concept score, a response to the first case.

14. The system of claim 9, wherein the concept score is further computed based on the presence of a relationship between the first concept in the first case and a second concept in the in the candidate answer.

15. The system of claim 9, the operation further comprising detecting the first concept in the case.

16. The system of claim 9, wherein the first concept is stored in an ontology comprising a plurality of concepts and concept relationships.