Tailoring Question Answering System Output Based on User Expertise

• US 20160048772A1
• Filed: 08/14/2014
• Published: 02/18/2016
• Est. Priority Date: 08/14/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Byron, Donna K., Bruno, Nicholas V., Goth, Julius III, Mansjur, Dwi Sianto
• Granted Patent: US 10,366,332 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

A mechanism is provided in a data processing system for tailoring question answering system output based on user expertise. The mechanism receives an input question from a questioning user and determines a set of features associated with text of the input question. The mechanism determines an expertise level of the questioning user based on the set of features associated with the text of the input question using a trained expertise model. The mechanism generates one or more candidate answers for the input question and tailors output of the one or more candidate answers based on the expertise level of the questioning user.

Claims

1. A method, in a data processing system, for tailoring question answering system output based on user expertise, the method comprising:

receiving, by the data processing system, an input question from a questioning user;

determining, by the data processing system, a set of features associated with text of the input question;

determining, by the data processing system, an expertise level of the questioning user based on the set of features associated with the text of the input question using a trained expertise model;

generating, by the data processing system, one or more candidate answers for the input question; and

tailoring, by the data processing system, output of the one or more candidate answers based on the expertise level of the questioning user.

2. The method of claim 1, wherein determining the set of features associated with the text of the input question comprises:

extracting a plurality of features from the text of the input question using an annotation engine pipeline in the data processing system.

3. The method of claim 2, wherein the plurality of features comprises at least one of content words formed into unigram/ngram lexical features, social hedges, specificity of words, specific experience level indicators, or references to external expertise.

4. The method of claim 2, wherein determining the set of features associated with the text of the input question further comprises:

obtaining features from the questioning user'"'"'s posting history within a collection of question and answer postings.

5. The method of claim 4, wherein determining the set of features associated with the text of the input question further comprises:

obtaining features from responses by other users within the collection of question and answer postings.

6. The method of claim 1, wherein the trained expertise model comprises a question partition trained using questions in a collection of question and answer postings and an answer partition trained using answers in the collection of question and answer postings and wherein determining the expertise level of the questioning user comprises determining the expertise level of the questioning user using the question partition of the trained expertise model.

7. The method of claim 1, wherein generating the one or more candidate answers for the input question comprises generating the one or more candidate answers from a collection of question and answer postings.

8. The method of claim 7, wherein tailoring output of the one or more candidate answers comprises:

determining an expertise level of a contributing user providing evidence for a given candidate answer, comprising;

obtaining features from the contributing user'"'"'s posting history within the collection of question and answer postings; and

obtaining features from responses by other users within the collection of question and answer postings.

9. The method of claim 1, wherein tailoring output of the one or more candidate answers comprises:

determining an expertise level of each of the one or more candidate answers using the trained expertise model; and

ranking the one or more candidate answers based on the expertise levels of the one or more candidate answers.

10. The method of claim 1, wherein tailoring output of the one or more candidate answers comprises:

selecting only candidate answers that have a high confidence score and match the expertise level of the questioning user.

11. The method of claim 1, wherein training the trained expertise model comprises:

harvesting a collection of question and answer postings;

labeling questions and answers in the collection with predetermined expertise levels;

determining a set of features associated with text of each question and answer; and

training a machine learning model based on the predetermined expertise levels and the sets of features associated with the text of the questions and answers to form the trained expertise model.

12. The method of claim 11, wherein determining the set of features associated with text of a given question or answer comprises:

extracting a plurality of features from the text of the given question or answer using an annotation engine pipeline;

obtaining features from posting history of a contributing user associated with the given question or answer; and

obtaining features from responses by other users within the collection of question and answer postings.

13. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

receive an input question from a questioning user;

determine a set of features associated with text of the input question;

determine an expertise level of the questioning user based on the set of features associated with the text of the input question using a trained expertise model;

generate one or more candidate answers for the input question; and

tailor output of the one or more candidate answers based on the expertise level of the questioning user.

14. The computer program product of claim 13, wherein determining the set of features associated with the text of the input question comprises:

extracting a plurality of features from the text of the input question using an annotation engine pipeline.

15. The computer program product of claim 13, wherein tailoring output of the one or more candidate answers comprises:

determining an expertise level of each of the one or more candidate answers using the trained expertise model; and

ranking the one or more candidate answers based on the expertise levels of the one or more candidate answers.

16. The computer program product of claim 13, wherein tailoring output of the one or more candidate answers comprises:

selecting only candidate answers that have a high confidence score and match the expertise level of the questioning user.

17. The computer program product of claim 13, wherein training the trained expertise model comprises:

harvesting a collection of question and answer postings;

labeling questions and answers in the collection with predetermined expertise levels;

determining a set of features associated with text of each question and answer; and

training a machine learning model based on the predetermined expertise levels and the sets of features associated with the text of the questions and answers to form the trained expertise model.

18. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

receive an input question from a questioning user;

determine a set of features associated with text of the input question;

determine an expertise level of the questioning user based on the set of features associated with the text of the input question using a trained expertise model;

generate one or more candidate answers for the input question; and

tailor output of the one or more candidate answers based on the expertise level of the questioning user.

19. The apparatus of claim 18, wherein tailoring output of the one or more candidate answers comprises:

determining an expertise level of each of the one or more candidate answers using the trained expertise model; and

ranking the one or more candidate answers based on the expertise levels of the one or more candidate answers.

20. The apparatus of claim 18, wherein training the trained expertise model comprises:

harvesting a collection of question and answer postings;

labeling questions and answers in the collection with predetermined expertise levels;

determining a set of features associated with text of each question and answer; and

training a machine learning model based on the predetermined expertise levels and the sets of features associated with the text of the questions and answers to form the trained expertise model.