Question resolution processing in deep question answering systems

• US 10,062,032 B2
• Filed: 06/17/2014
• Issued: 08/28/2018
• Est. Priority Date: 02/20/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Allen, Corville O., Clark, Adam T., Huebert, Jeffrey K., Payton, Aspen L., Petri, John E.
• Primary Examiner(s): Chen, Alan
• Application Number:
  US14/306,464
• Publication Number:
  US 20150234825A1
• Time in Patent Office: 1,533 Days
• Field of Search: None
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G16H 10/20 for electronic clinical trials or questionnaires

Abstract

System, method, and computer program product for performing an operation, the operation comprising receiving, by a deep question answering system, a question not specifying an element of input data, identifying a set of possible values for the element of input data, generating, by the deep question answering system, a respective set of candidate answers for the question: (i) without the element of input data, and (ii) with each possible value in the set of possible values for the element of input data, and returning at least one candidate answer from at least one set of candidate answers as responsive to the question.

Claims

1. A computer-implemented method, comprising:

receiving, by an interface of a question answering (QA) system, a question not specifying a first element of input data, of a plurality of elements of input data specified in the question;

determining, based on feature classification data generated during a training phase of the QA system, that specifying one possible value of the set of possible values for the first element of input data in the question allows the QA system to;

(i) increase a confidence score of at least one candidate answer generated for the question without the value for the first element of input data beyond a specified confidence threshold, and (ii) identify an item of supporting evidence that is not identifiable without the value for the first element of input data;

identifying a set of possible values for the first element of input data, wherein the set of possible values is a finite set of possible values including a first possible value and a second possible value;

generating, by the QA system, a plurality of sets of candidate answers for the question, the plurality of sets of candidate answers comprising;

(i) a first set of candidate answers generated without using any of the possible values for the first element of input data, (ii) a second set of candidate answers generated using the first possible value for the first element of input data, (iii) a third set of candidate answers generated using the second possible value for the first element of input data, and (iv) a respective set of candidate answers generated using each remaining possible value in the set of possible values for the first element of input data; and

returning at least one candidate answer having a confidence score exceeding the specified confidence threshold from at least one set of candidate answers as responsive to the question.

2. The method of claim 1, wherein the set of possible values for the first element of input data are identified in a corpus of information, wherein the feature classification data comprises a mapping between the first element of input data and a class of questions.

3. The method of claim 2, further comprising:

prior to generating the respective sets of candidate answers;

determining that a count of the possible values in the set of possible values for the first element of input data does not exceed a maximum threshold; and

determining that the received question is of the first class of questions.

4. The method of claim 1, further comprising:

determining, based on the feature classification data, that at least one of the set of possible values for the first element of input data, allows the QA system to identify a candidate answer that is not identifiable when the value for the first element of input data is not included in the question.

5. The method of claim 1, further comprising:

subsequent to returning the at least one candidate answer, receiving by the interface a value for the first element of input data from a user.

6. The method of claim 1, wherein returning at least one candidate answer from at least one of the sets of candidate answers as responsive to the question comprises:

determining that a first candidate answer included in each respective set of candidate answers generated for the question with each possible value in the set of possible values for the first element of input data, has a confidence score greater than a confidence score for all other candidate answers in each respective set of candidate answers; and

returning the first candidate answer as responsive to the question.

7. The method of claim 1, wherein a subset of the candidate answers of a subset of the respective sets of candidate answers are returned as responsive to the question.

8. The method of claim 1, wherein the QA system generates the feature classification data during the training phase by processing a plurality of other questions prior to receiving the question from the interface, wherein the training phase generates a machine learning model specifying the feature classification data.