Question classification and feature mapping in a deep question answering system

• US 9,754,215 B2
• Filed: 12/17/2012
• Issued: 09/05/2017
• Est. Priority Date: 12/17/2012
• Status: Active Grant

• Current Assignee: SINOEAST CONCEPT LIMITED (Tencent Holdings Limited)
• Original Assignee: SINOEAST CONCEPT LIMITED (Tencent Holdings Limited)
• Inventors: Clark, Adam T., Megerian, Mark G., Petri, John E., Stevens, Richard J.
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): LAMARDO, VIKER ALEJANDRO
• Application Number:
  US13/717,043
• Publication Number:
  US 20140172756A1
• Time in Patent Office: 1,723 Days
• Field of Search: 706 12
• CPC Class Codes: G06F 17/00 Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06Q 10/00 Administration; Management

Abstract

System, method, and computer program product to identify relevant features in a deep question answering system, by classifying a first case received by the deep question answering system, and, while training the deep question answering system to answer the first case, identifying a first feature in the first case, computing a first feature score for the first feature, the first feature score indicating a relevance of the first feature in generating a correct response to the first case, and, identifying the first feature as relevant in answering the classified first case upon determining that the first feature score exceeds a relevance threshold.

Claims

1. A computer program product to identify features by a question answering system, the computer program product comprising:

a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by a processor to perform an operation comprising;

classifying a first case received by the question answering system as being of a first type; and

training the question answering system to generate a response to the first case, wherein the training comprises;

identifying a first feature and identifying a first feature and a second feature of the first case, wherein the first and second features comprise a first and a second variable of the first case, respectively;

determining that the question answering system identified evidence based on the first and second features of the first case;

identifying a set of possible values for the first and second variables in the identified evidence, wherein the set of possible values for the first and second variables include one or more of;

(i) a first value comprising reference value for the respective variables, (ii) a second value comprising an actual value for the respective variables, and (iii) a third value comprising an opinion value for the respective variables;

determining that the question answering system generated the response to the first case based on the identified evidence and a weight applied to a combined feature comprising the first and second features, wherein the weight is of a range of weights applied by the question answering system to the combined feature;

computing a first feature score for the first feature, wherein the first feature score indicates the weight applied by the question answering system to the combined feature and the sets of possible values for the first and second variables when generating the response to the first case; and

storing an indication of a relationship between the first feature, the second feature, the combined feature, the identified evidence, and cases classified as being of the first type upon determining that the first feature score exceeds a threshold, wherein the relationship reflects the weight applied by the question answering system to the combined feature when generating responses to cases classified as being of the first type;

wherein the operation further comprises;

subsequent to training the question answering system and responsive to receiving a second case by the question answering system;

computing a similarity score for the first case and the second case;

upon determining that the similarity score exceeds a specified similarity threshold, classifying the second case as being of the first type; and

while generating a response to the second case;

refraining from processing a first candidate answer to determine whether to return the first candidate answer as responsive to the second case upon determining the first candidate answer does not include the combined feature, thereby reducing an amount of time and processing resources required to generate a response to the second case relative to the amount of time and processing resources required to generate a response to the second case by processing the first candidate answer; and

processing a second candidate answer to determine whether to return the second candidate answer as responsive to the second case upon determining the second candidate answer includes the combined feature.

2. The computer program product of claim 1, wherein the feature score is computed by determining a count of words appearing in a text of the first case and a text of the first candidate answer, wherein the similarity score is based on each of:

(i) the classification of the first case and the classification second case, and (ii) a context of the first case and a context of the second case.

3. The computer program product of claim 2, wherein the first case and the second case are further classified based on each of:

(i) metadata describing an attribute of a user presenting the respective case to the question answering system, (ii) a content of a question in the respective case, and (iii) a type of the question in the respective case, wherein the metadata is received with the first case.

4. The computer program product of claim 3, wherein the attribute comprises:

(i) a role of the user, (ii) an affiliation of the user, (iii) an expertise of the user, and (iv) a set of predefined preferences of the user.

5. The computer program product of claim 1, wherein the first feature score is further based on a number of items of supporting evidence identified by the question answering system using the first and second features when answering the first case.

6. The computer program product of claim 1, wherein the first case comprises at least one question presented to the question answering system.

14. The computer program product of claim 1, the operation further comprising:

determining that the question answering system did not generate the response to the first case based on a third feature;

computing a second feature score for the third feature, wherein the second feature score indicates a weight applied by the question answering system to the third feature when generating the response to the first case;

determining that the second feature score does not exceed the threshold; and

storing an indication to refrain from processing a candidate answer including the third feature when processing a second case received by the question answering system.

7. A system, comprising:

one or more computer processors; and

a memory containing a program, which, when executed by the one or more computer processors, performs an operation to identify features by a question answering system, the operation comprising;

training the question answering system to generate a response to the first case, wherein the training comprises;

identifying a first feature and a second feature of the first case, wherein the first and second features comprise a first and a second variable of the first case, respectively;

determining that the question answering system identified evidence based on the first and second features of the first case;

identifying a set of possible values for the first and second variables in the identified evidence, wherein the set of possible values for the first and second variables include one or more of;

(i) a first value comprising reference value for the respective variables, (ii) a second value comprising an actual value for the respective variables, and (iii) a third value comprising an opinion value for the respective variables;

determining that the question answering system generated the response to the first case based on the identified evidence and a weight applied to a combined feature comprising the first and second features, wherein the weight is of a range of weights applied by the question answering system to the combined feature;

computing a first feature score for the first feature, wherein the first feature score indicates the weight applied by the question answering system to the combined feature and the sets of possible values for the first and second variables when generating the response to the first case; and

storing an indication of a relationship between the first feature, the second feature, the combined feature, the identified evidence, and cases classified as being of the first type upon determining that the first feature score exceeds a threshold, wherein the relationship reflects the weight applied by the question answering system to the combined feature when generating responses to cases classified as being of the first type;

wherein the operation further comprises;

subsequent to training the question answering system and responsive to receiving a second case by the question answering system;

computing a similarity score for the first case and the second case;

upon determining that the similarity score exceeds a specified similarity threshold, classifying the second case as being of the first type; and

while generating a response to the second case;

refraining from processing a first candidate answer to determine whether to return the first candidate answer as responsive to the second case upon determining the first candidate answer does not include the combined feature, thereby reducing an amount of time and processing resources required to generate a response to the second case relative to the amount of time and processing resources required to generate a response to the second case by processing the first candidate answer; and

processing a second candidate answer to determine whether to return the second candidate answer as responsive to the second case upon determining the second candidate answer includes the combined feature.

8. The system of claim 7, wherein the feature score is computed by determining a count of words appearing in a text of the first case and a text of the first candidate answer, wherein the similarity score is based on each of:

(i) the classification of the first case and the classification second case, and (ii) a context of the first case and a context of the second case.

9. The system of claim 8, wherein the first case and the second case are further classified based on each of:

(i) metadata describing an attribute of a user presenting the respective case to the question answering system, (ii) a content of a question in the respective case, and (iii) a type of the question in the respective case, wherein the metadata is received with the first case.

10. The system of claim 9, wherein the attribute comprises:

(i) a role of the user, (ii) an affiliation of the user, (iii) an expertise of the user, and (iv) a set of predefined preferences of the user.

11. The system of claim 7, wherein the first feature score is further based on a number of items of supporting evidence identified by the question answering system using the first and second features when answering the first case.

12. The system of claim 7, wherein the first case comprises at least one question presented to the question answering system.

13. The system of claim 7, the operation further comprising:

determining that the question answering system did not generate the response to the first case based on a third feature;

computing a second feature score for the third feature, wherein the second feature score indicates a weight applied by the question answering system to the third feature when generating the response to the first case;

determining that the second feature score does not exceed the threshold; and

storing an indication to refrain from processing a candidate answer including the third feature when processing a second case received by the question answering system.