Tailoring question answering system output based on user expertise

• US 10,366,332 B2
• Filed: 08/14/2014
• Issued: 07/30/2019
• Est. Priority Date: 08/14/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bruno, Nicholas V., Byron, Donna K., Goth, III, Julius, Mansjur, Dwi Sianto
• Primary Examiner(s): Zhen, Li B.
• Assistant Examiner(s): Buss, Benjamin J
• Application Number:
  US14/460,066
• Publication Number:
  US 20160048772A1
• Time in Patent Office: 1,811 Days
• Field of Search: None
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

A mechanism is provided in a data processing system for tailoring question answering system output based on user expertise. The mechanism receives an input question from a questioning user and determines a set of features associated with text of the input question. The mechanism determines an expertise level of the questioning user based on the set of features associated with the text of the input question using a trained expertise model. The mechanism generates one or more candidate answers for the input question and tailors output of the one or more candidate answers based on the expertise level of the questioning user.

Claims

1. A method, in a data processing system comprising at least one processor and at least one memory, the at least one memory comprising instructions which are executed by the at least one processor and configure the processor to implement a question answering system for tailoring question answering system output based on user expertise, the method comprising:

training an expertise model, comprising;

harvesting a collection of question and answer postings;

labeling questions and answers in the collection with predetermined expertise levels;

determining a set of features associated with text of each question and answer; and

training a machine learning model based on the predetermined expertise levels and the sets of features associated with the text of the questions and answers to form the trained expertise model, wherein the trained expertise model comprises a question partition trained using questions in the collection of question and answer postings and an answer partition trained using answers in the collection of question and answer postings;

receiving, by the question answering system executing a question answering pipeline on the at least one processor of the data processing system, an input question from a questioning user,

determining, by a question and topic analysis stage of the question answering pipeline, a set of features associated with text of the input question, wherein determining the set of features associated with the text of the input question comprises extracting a plurality of features from the text of the input question using an annotation engine pipeline in the data processing system;

obtaining features from the questioning user'"'"'s posting history within a collection of question and answer postings, wherein the features from the questioning user'"'"'s posting history include a percentage of the questioning user'"'"'s posts that are questions versus answers;

determining, by the question answering pipeline, an expertise level of the questioning user based on the set of features associated with the text of the input question and based on at least the percentage of the questioning user'"'"'s posts that are questions versus answers using the question partition of the trained expertise model;

generating, by a hypothesis generation stage of the question answering pipeline, one or more candidate answers for the input question; and

tailoring, by the question answering system, output of the one or more candidate answers based on the expertise level of the questioning user.

2. The method of claim 1, wherein the plurality of features comprises at least one of content words formed into unigram/ngram lexical features, social hedges, specificity of words, specific experience level indicators, or references to external expertise.

3. The method of claim 1, wherein obtaining the features from the questioning user'"'"'s posting history further comprises:

obtaining features from responses by other users to the questioning user'"'"'s posting history within the collection of question and answer postings, wherein the features from responses by other users comprise a number of consent indicators that indicate whether the other users found an answer provided by the questioning user helpful.

4. The method of claim 1, wherein generating the one or more candidate answers for the input question comprises generating the one or more candidate answers from the collection of question and answer postings.

5. The method of claim 4, wherein tailoring output of the one or more candidate answers comprises:

determining an expertise level of a contributing user providing evidence for a given candidate answer, comprising;

obtaining features from the contributing user'"'"'s posting history within the collection of question and answer postings; and

obtaining features from responses by other users within the collection of question and answer postings.

6. The method of claim 1, wherein tailoring output of the one or more candidate answers comprises:

determining an expertise level of each of the one or more candidate answers using the trained expertise model; and

ranking the one or more candidate answers based on the expertise levels of the one or more candidate answers.

7. The method of claim 1, wherein tailoring output of the one or more candidate answers comprises:

selecting only candidate answers that have a high confidence score and match the expertise level of the questioning user.

8. The method of claim 1, wherein determining the set of features associated with text of a given question or answer comprises:

extracting a plurality of features from the text of the given question or answer using an annotation engine pipeline;

obtaining features from posting history of a contributing user associated with the given question or answer, and

obtaining features from responses by other users within the collection of question and answer postings.

9. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to implement a question answering system for tailoring question answering system output based on user expertise, wherein the computer readable program further causes the computing device to:

train an expertise model, comprising;

harvesting a collection of question and answer postings;

labeling questions and answers in the collection with predetermined expertise levels;

determining a set of features associated with text of each question and answer; and

training a machine learning model based on the predetermined expertise levels and the sets of features associated with the text of the questions and answers to form the trained expertise model, wherein the trained expertise model comprises a question partition trained using questions in the collection of question and answer postings and an answer partition trained using answers in the collection of question and answer postings;

receive, by the question answering system executing a question answering pipeline on at least one processor of the computing device, an input question from a questioning user;

determine, by a question and topic analysis stage of the question answering pipeline, a set of features associated with text of the input question, wherein determining the set of features associated with the text of the input question comprises extracting a plurality of features from the text of the input question using an annotation engine pipeline in the data processing system;

obtain features from the questioning user'"'"'s posting history within a collection of question and answer postings, wherein the features from the questioning user'"'"'s posting history include a percentage of the questioning user'"'"'s posts that are questions versus answers;

determine, by the question answering pipeline, an expertise level of the questioning user based on the set of features associated with the text of the input question and based on at least the percentage of the questioning user'"'"'s posts that are questions versus answers using the question partition of the trained expertise model;

generate, by a hypothesis generation stage of the question answering pipeline, one or more candidate answers for the input question; and

tailor, by the question answering system, output of the one or more candidate answers based on the expertise level of the questioning user.

10. The computer program product of claim 9, wherein tailoring output of the one or more candidate answers comprises:

determining an expertise level of each of the one or more candidate answers using the trained expertise model; and

ranking the one or more candidate answers based on the expertise levels of the one or more candidate answers.

11. The computer program product of claim 9, wherein tailoring output of the one or more candidate answers comprises:

selecting only candidate answers that have a high confidence score and match the expertise level of the questioning user.

12. The computer program product of claim 9, wherein obtaining the features from the questioning user'"'"'s posting history further comprises:

obtaining features from responses by other users to the questioning user'"'"'s posting history within the collection of question and answer postings, wherein the features from responses by other users comprise a number of consent indicators that indicate whether the other users found an answer provided by the questioning user helpful.

13. The computer program product of claim 9, wherein generating the one or more candidate answers for the input question comprises generating the one or more candidate answers from a collection of question and answer postings.

14. The computer program product of claim 13, wherein tailoring output of the one or more candidate answers comprises:

determining an expertise level of a contributing user providing evidence for a given candidate answer, comprising;

obtaining features from the contributing user'"'"'s posting history within the collection of question and answer postings; and

obtaining features from responses by other users within the collection of question and answer postings.

15. The computer program product of claim 9, wherein determining the set of features associated with text of a given question or answer comprises:

extracting a plurality of features from the text of the given question or answer using an annotation engine pipeline;

obtaining features from posting history of a contributing user associated with the given question or answer; and

obtaining features from responses by other users within the collection of question and answer postings.

16. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to implement a question answering system for tailoring question answering system output based on user expertise, wherein the instructions further cause the processor to;

train an expertise model, comprising;

harvesting a collection of question and answer postings;

labeling questions and answers in the collection with predetermined expertise levels;

determining a set of features associated with text of each question and answer, and

training a machine learning model based on the predetermined expertise levels and the sets of features associated with the text of the questions and answers to form the trained expertise model, wherein the trained expertise model comprises a question partition trained using questions in the collection of question and answer postings and an answer partition trained using answers in the collection of question and answer postings;

receive, by the question answering system executing a question answering pipeline on the processor, an input question from a questioning user;

determine, by a question and topic analysis stage of the question answering pipeline, a set of features associated with text of the input question, wherein determining the set of features associated with the text of the input question comprises extracting a plurality of features from the text of the input question using an annotation engine pipeline in the data processing system;

obtain features from the questioning user'"'"'s posting history within a collection of question and answer postings, wherein the features from the questioning user'"'"'s posting history include a percentage of the questioning user'"'"'s posts that are questions versus answers;

determine, by the question answering pipeline, an expertise level of the questioning user based on the set of features associated with the text of the input question and based on at least the percentage of the questioning user'"'"'s posts that are questions versus answers using the question partition of the trained expertise model;

generate, by a hypothesis generation stage of the question answering pipeline, one or more candidate answers for the input question; and

tailor, by the question answering system, output of the one or more candidate answers based on the expertise level of the questioning user.

17. The apparatus of claim 16, wherein tailoring output of the one or more candidate answers comprises:

determining an expertise level of each of the one or more candidate answers using the trained expertise model; and

ranking the one or more candidate answers based on the expertise levels of the one or more candidate answers.