Question classification and feature mapping in a deep question answering system

• US 9,911,082 B2
• Filed: 02/18/2013
• Issued: 03/06/2018
• Est. Priority Date: 12/17/2012
• Status: Active Grant

• Current Assignee: SINOEAST CONCEPT LIMITED (Tencent Holdings Limited)
• Original Assignee: SINOEAST CONCEPT LIMITED (Tencent Holdings Limited)
• Inventors: Clark, Adam T., Megerian, Mark G., Petri, John E., Stevens, Richard J.
• Primary Examiner(s): Lewis, David L
• Assistant Examiner(s): WILLIAMS, ROSS A
• Application Number:
  US13/769,409
• Publication Number:
  US 20140172139A1
• Time in Patent Office: 1,842 Days
• Field of Search: 434156-166, 434322, 434350-364, 700 92, 706 45, 706 46
• CPC Class Codes: G06F 17/00 Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06Q 10/00 Administration; Management

Abstract

System, method, and computer program product to identify relevant features in a deep question answering system, by classifying a first case received by the deep question answering system, and, while training the deep question answering system to answer the first case, identifying a first feature in the first case, computing a first feature score for the first feature, the first feature score indicating a relevance of the first feature in generating a correct response to the first case, and, identifying the first feature as relevant in answering the classified first case upon determining that the first feature score exceeds a relevance threshold.

Claims

1. A method, comprising:

classifying a first case received by the question answering system as being of a first type; and

training the question answering system to generate a response to the first case, wherein the training comprises;

identifying a first feature and a second feature of the first case, wherein the first and second features comprise a first and a second variable of the first case, respectively;

determining that the question answering system identified evidence while generating the response to the first case based on the first and second features of the first case;

identifying a set of possible values for the first and second variables in the identified evidence, wherein the set of possible values for the first and second variables include;

(i) a first value comprising reference value for the respective variables, (ii) a second value comprising an actual value for the respective variables, and (iii) a third value comprising an opinion value for the respective variables;

determining that the question answering system generated the response to the first case based on the identified evidence and a weight applied to a combined feature comprising the first and second features, wherein the weight is of a range of weights applied by the question answering system to the combined feature;

computing a first feature score for the combined feature, wherein the first feature score indicates the weight applied by the question answering system to the combined feature and the sets of possible values for the first and second variables when generating the response to the first case; and

storing an indication of a relationship between the first feature, the second feature, the combined feature, the identified evidence, and cases classified as being of the first type upon determining that the first feature score exceeds a threshold, wherein the relationship reflects the weight applied to the combined feature when generating responses to cases classified as being of the first type.

2. The method of claim 1, further comprising:

subsequent to training the question answering system and responsive to receiving a second case by the question answering system;

computing a similarity score for the first case and the second case;

upon determining that the similarity score exceeds a specified similarity threshold, classifying the second case as being of the first type; and

while generating a response to the second case;

refraining from processing a first candidate answer to determine whether to return the first candidate answer as responsive to the second case upon determining the first candidate answer does not include the combined feature; and

processing a second candidate answer to determine whether to return the second candidate answer as responsive to the second case upon determining the second candidate answer includes the combined feature.

3. The method of claim 2, wherein the feature score is computed by determining a count of words appearing in a text of the first case and a text of the first candidate answer, wherein the similarity score is based on each of:

(i) the classification of the first case and the classification of the second case, and (ii) a context of the first case and a context of the second case.

4. The method of claim 3, wherein the first case and the second case are further classified based on each of:

(i) metadata describing an attribute of a user presenting the respective case to the question answering system, (ii) a content of a question in the respective case, and (iii) a type of the question in the respective case, wherein the metadata is received with the first case.

5. The method of claim 4, wherein the attribute comprises:

(i) a role of the user, (ii) an affiliation of the user, (iii) an expertise of the user, and (iv) a set of predefined preferences of the user.

6. The method of claim 1, wherein the first feature score is further based on a number of relevant items of supporting evidence retrieved by the question answering system using the first and second features.

8. The method of claim 1, wherein the first case comprises at least one question presented to the question answering system.

7. A system, comprising:

one or more computer processors; and

a memory containing a program, which, when executed by the one or more computer processors, performs an operation to identify features by a question answering system, the operation comprising;

classifying a first case received by the question answering system as being of a first type; and

training the question answering system to generate a response to the first case, wherein the training comprises;

identifying a first feature and a second feature of the first case, wherein the first and second features comprise a first and a second variable of the first case, respectively;

determining that the question answering system identified evidence while generating the response to the first case based on the first and second features of the first case;

identifying a set of possible values for the first and second variables in the identified evidence, wherein the set of possible values for the first and second variables include;

(i) a first value comprising reference value for the respective variables, (ii) a second value comprising an actual value for the respective variables, and (iii) a third value comprising an opinion value for the respective variables;

determining that the question answering system generated the response to the first case based on the identified evidence and a weight applied to a combined feature comprising the first and second features, wherein the weight is of a range of weights applied by the question answering system to the combined feature;

computing a first feature score for the combined feature, wherein the first feature score indicates the weight applied by the question answering system to the combined feature and the sets of possible values for the first and second variables when generating the response to the first case; and

storing an indication of a relationship between the first feature, the second feature, the combined feature, the identified evidence, and cases classified as being of the first type upon determining that the first feature score exceeds a threshold, wherein the relationship reflects the weight applied to the combined feature when generating responses to cases classified as being of the first type.

9. The system of claim 7, wherein the operation further comprises:

subsequent to training the question answering system and responsive to receiving a second case by the question answering system;

computing a similarity score for the first case and the second case;

upon determining that the similarity score exceeds a specified similarity threshold, classifying the second case as being of the first type; and

while generating a response to the second case;

refraining from processing a first candidate answer to determine whether to return the first candidate answer as responsive to the second case upon determining the first candidate answer does not include the combined feature; and

processing a second candidate answer to determine whether to return the second candidate answer as responsive to the second case upon determining the second candidate answer includes the combined feature.

10. The system of claim 9, wherein the feature score is computed by determining a count of words appearing in a text of the first case and a text of the first candidate answer, wherein the similarity score is based on each of:

(i) the classification of the first case and the classification of the second case, and (ii) a context of the first case and a context of the second case.

11. The system of claim 10, wherein the first case and the second case are further classified based on each of:

(i) metadata describing an attribute of a user presenting the respective case to the question answering system, (ii) a content of a question in the respective case, and (iii) a type of the question in the respective case, wherein the metadata is received with the first case.

12. The system of claim 11, wherein the attribute comprises:

(i) a role of the user, (ii) an affiliation of the user, (iii) an expertise of the user, and (iv) a set of predefined preferences of the user.

13. The system of claim 7, wherein the first feature score is further based on a number of relevant items of supporting evidence retrieved by the question answering system using the first and second features.

14. The system of claim 7, wherein the first case comprises at least one question presented to the question answering system.