Question resolution processing in deep question answering systems

• US 10,068,175 B2
• Filed: 02/20/2014
• Issued: 09/04/2018
• Est. Priority Date: 02/20/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Allen, Corville O., Clark, Adam T., Huebert, Jeffrey K., Payton, Aspen L., Petri, John E.
• Primary Examiner(s): Chen, Alan
• Application Number:
  US14/185,148
• Publication Number:
  US 20150235131A1
• Time in Patent Office: 1,657 Days
• Field of Search: None
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G16H 10/20 for electronic clinical trials or questionnaires

Abstract

System, method, and computer program product for performing an operation, the operation comprising receiving, by a deep question answering system, a question not specifying an element of input data, identifying a set of possible values for the element of input data, generating, by the deep question answering system, a respective set of candidate answers for the question: (i) without the element of input data, and (ii) with each possible value in the set of possible values for the element of input data, and returning at least one candidate answer from at least one set of candidate answers as responsive to the question.

Claims

1. A computer program product, comprising:

a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by a processor to perform an operation comprising;

receiving, by an interface of a question answering (QA) system, a question not specifying a value for a first element of input data, of a plurality of elements of input data specified in the question;

determining, based on feature classification data generated during a training phase of the QA system, that one possible value of the set of possible values for the first element of input data, if specified in the question, allows the question answering system to increase a confidence score of at least one candidate answer generated for the question without the value for the first element of input data, beyond a specified confidence threshold;

identifying a set of possible values for the first element of input data, wherein the set of possible values is a finite set of possible values including a first possible value and a second possible value;

generating, by the question answering system, a respective set of candidate answers for the question;

(i) without using any of the possible values for the first element of input data, (ii) using the first possible value for the first element of input data, (iii) using the second possible value for the first element of input data, and (iv) using each remaining possible value in the set of possible values for the first element of input data; and

returning at least one candidate answer from at least one of the sets of candidate answers as responsive to the question.

2. The computer program product of claim 1, wherein the set of possible values for the first element of input data are identified from a corpus of information.

3. The computer program product of claim 1, the operation further comprising:

prior to generating the respective sets of candidate answers, determining that a count of the possible values in the set of possible values for the first element of input data does not exceed a maximum threshold.

4. The computer program product of claim 1, the operation further comprising:

determining that at least one of the set of possible values for the first element of input data, allows the question answering system to identify an item of supporting evidence that is not identified when the value for the first element of input data is not included in the question.

5. The computer program product of claim 1, the operation further comprising:

subsequent to returning the at least one candidate answer, providing an interface configured to receive, from a user, a correct value for the first element of input data.

6. The computer program product of claim 1, wherein returning at least one candidate answer from at least one of the sets of candidate answers as responsive to the question comprises:

determining that a first candidate answer included in each respective set of candidate answers generated for the question with each possible value in the set of possible values for the first element of input data, has a confidence score greater than a confidence score for all other candidate answers in each respective set of candidate answers; and

returning the first candidate answer as responsive to the question.

7. The computer program product of claim 1, wherein a subset of the candidate answers of a subset of the respective sets of candidate answers are returned as responsive to the question.

11. The computer program product of claim 1, wherein the question answering system generates the feature classification data during the training phase by processing a plurality of other questions prior to receiving the question from the interface, wherein the training phase generates a machine learning model specifying the feature classification data.

8. A system, comprising:

one or more computer processors; and

a memory containing a program, which, when executed by the one or more computer processors, performs an operation comprising;

receiving, by an interface of a question answering (QA) system, a question not specifying a value for a first element of input data, of a plurality of elements of input data specified in the question;

determining, based on feature classification data generated during a training phase of the QA system, that one possible value of the set of possible values for the first element of input data, if specified in the question, allows the question answering system to increase a confidence score of at least one candidate answer generated for the question without the value for the first element of input data, beyond a specified confidence threshold;

identifying a set of possible values for the first element of input data, wherein the set of possible values is a finite set of possible values including a first possible value and a second possible value;

generating, by the question answering system, a respective set of candidate answers for the question;

(i) without using any of the possible values for the first element of input data, (ii) using the first possible value for the first element of input data, (iii) using the second possible value for the first element of input data, and (iv) using each remaining possible value in the set of possible values for the first element of input data; and

returning at least one candidate answer from at least one of the sets of candidate answers as responsive to the question.

9. The system of claim 8, wherein the set of possible values for the first element of input data are identified in a corpus of information.

10. The system of claim 8, wherein candidate answers from each set of candidate answers are returned, the operation further comprising:

receiving, from a user, a correct value for the first element of input data, wherein the correct value is a member of the set of possible values for the first element of input data; and

returning at least one candidate answer from the sets of candidate answers using the correct value for the first element of input data as responsive to the question without re-processing the question to generate a response using the correct value.

12. The system of claim 8, wherein the question answering system generates the feature classification data during the training phase by processing a plurality of other questions prior to receiving the question from the interface, wherein the training phase generates a machine learning model specifying the feature classification data.