Knowledge corroboration

• US 8,706,653 B2
• Filed: 12/08/2010
• Issued: 04/22/2014
• Est. Priority Date: 12/08/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kasneci, Gjergji, Van Gael, Jurgen Anne Francois Marie, Herbrich, Ralf, Stern, David, Kraepel, Thore
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Wu, Fuming
• Application Number:
  US12/963,352
• Publication Number:
  US 20120150771A1
• Time in Patent Office: 1,231 Days
• Field of Search: None
• US Class Current: 706/12
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

Knowledge corroboration is described. In an embodiment many judges provide answers to many questions so that at least one answer is provided to each question and at least some of the questions have answers from more than one judge. In an example a probabilistic learning system takes features describing the judges or the questions or both and uses those features to learn an expertise of each judge. For example, the probabilistic learning system has a graphical assessment component which aggregates the answers in a manner which takes into account the learnt expertise in order to determine enhanced answers. In an example the enhanced answers are used for knowledge base clean-up or web-page classification and the learnt expertise is used to select judges for future questions. In an example the probabilistic learning system has a logical component that propagates answers according to logical relations between the questions.

Claims

1. A method of obtaining enhanced answers to a plurality of questions comprising:

accessing features comprising any one or more of;

at least one feature for each of a plurality of judges, and at least one feature for each of the questions;

accessing at least one answer for each question given by one of the judges and, for a plurality of the questions, more than one answer given by different ones of the judges;

using a probabilistic learning system to learn an expertise of each judge using the features, the probabilistic learning system using at least a first question for which an answer is known as ground truth and a second question for which a ground truth answer is unknown, true answers to at least some of the questions being unknown to the probabilistic learning system and true expertise of the judges for the questions being unknown to the probabilistic learning system, an enhanced answer to a question being more likely to be accurate than a corresponding answer before application of the probabilistic learning system; and

using the probabilistic learning system to determine enhanced answers to the questions by aggregating the answers in a manner which takes into account the identified expertise of the judges by weighting answers of the judges based on the identified expertise of the judges.

2. A method as claimed in claim 1 wherein accessing features comprises accessing question features and judge features and wherein using the probabilistic learning system to identify an expertise of each judge comprises mapping the question features and judge features into a trait space which may be any of:

a one dimensional trait space, a two dimensional trait space, a trait space of higher dimension than two.

3. A method as claimed in claim 2 wherein mapping the judges and questions into a trait space is achieved using a linear mapping.

4. A method as claimed in claim 1 wherein using the probabilistic learning system to determine enhanced answers to the questions comprises using an assessment component of the probabilistic learning system having a graphical structure for aggregating the answers comprising, a node representing a learnt enhanced answer connected to, one node for each observed answer to be aggregated to the learnt enhanced answer, each of the nodes for the observed answers being connected to at least one node representing a learnt expertise indicator of a judge who gave the observed answer.

5. A method as claimed in claim 4 wherein using the probabilistic learning system to determine enhanced answers comprises taking into account cases in which judges guess correct answers to the questions.

6. A method as claimed in claim 4 which comprises using a node representing a learnt guessing probability connected to each of the nodes for the observed answers;

the learnt guessing probability representing a probability that, in the event that a judge does not know the answer to a question, the judge will guess the correct answer to that question.

7. A method as claimed in claim 4 which comprises using two nodes representing a learnt correctness of the judge who gave the answer, one of those nodes representing the probability that the judge reported the correct answer given the correct answer was true and the other node representing the probability that the judge reported the correct answer given the correct answer was false.

8. A method as claimed in claim 1 wherein using the probabilistic learning system to determine enhanced answers to the questions comprises propagating answers made by the judges through nodes in a logical component of the probabilistic learning system according to logical relations between the questions.

9. A method as claimed in claim 8 wherein at least one node in the logical component represents all missing deductions that could lead to a specified enhanced answer.

10. A method as claimed in claim 1 wherein using the probabilistic learning system comprises:

forming a factor graph data structure in memory with variable nodes storing statistics describing probability distributions representing learnt knowledge about expertise of the judges and about enhanced answers and factor nodes storing computational instructions; and

carrying out probabilistic inference using message passing over the factor graph by executing the computational instructions and storing the results at the variable nodes.

11. A method as claimed in claim 1 which comprises using the enhanced answers and the learnt expertise of the judges to update a knowledge base storing the questions and associated answers.

12. A method as claimed in claim 1 wherein the questions are documents to be classified into specified categories and wherein the answers are the specified categories;

and wherein the method comprises using the enhanced answers and the learnt expertise of the judges to update a knowledge base storing associations between the categories and references to the documents.

13. A method as claimed in claim 1 which comprises using the learnt expertise of the judges to select which of the judges to use for future questions.

14. A method as claimed in claim 1 which comprises using the learnt expertise of the judges to reward judges based on that learnt expertise.

15. A probabilistic learning system comprising:

one or more processors;

an input operable with the one or more processors and arranged to access features comprising any one or more of;

at least one feature for each of a plurality of judges, and at least one feature for each of the questions;

the input also arranged to access at least one answer for each question given by one of the judges and, for a plurality of the questions, more than one answer given by different ones of the judges;

using an inference engine arranged to carry out inference using a graphical structure of the probabilistic learning system to learn an expertise of each judge using the features and to determine enhanced answers to the questions by aggregating the answers in a manner which takes into account the identified expertise of the judges, the inference engine being configured use to at least a first question for which an answer is known as ground truth and a second question for which a ground truth answer is unknown to learn the expertise of each judge, true answers to at least some of the questions being unknown to the probabilistic learning system and true expertise of the judges for the questions being unknown to the probabilistic learning system, an enhanced answer to a question being more likely to be accurate than a corresponding answer before application of the probabilistic learning system.

16. A probabilistic learning system as claimed in claim 15 wherein the graphical structure comprises an assessment component an expertise component and a logical component with the expertise and logical components both connected to the assessment component.

17. A probabilistic learning system as claimed in claim 15 wherein the expertise component comprises nodes representing the features and nodes which map those features into a trait space.

18. A method of obtaining enhanced answers to a plurality of questions comprising:

accessing features comprising any one or more of;

at least one feature for each of a plurality of judges, and at least one feature for each of the questions;

accessing at least one answer for each question given by one of the judges and, for a plurality of the questions, more than one answer given by different ones of the judges;

using a probabilistic learning system to learn an expertise of each judge using the features; and

using the probabilistic learning system to determine enhanced answers to the questions by aggregating the answers in a manner which takes into account the identified expertise of the judges, the probabilistic learning system using at least a first question for which an answer is known as ground truth and a second question for which a ground truth answer is unknown, the at least first question being used to determine a reliablity of the judges, true answers to at least some of the questions being unknown to the probabilistic learning system and true expertise of the judges for the questions being unknown to the probabilistic learning system, an enhanced answer to a question being more likely to be accurate than a corresponding answer before application of the probabilistic learning system;

wherein using the probabilistic learning system to determine enhanced answers to the questions comprises using an assessment component of the probabilistic learning system having a graphical structure for aggregating the answers comprising, a node representing a learnt enhanced answer connected to, one node for each observed answer to be aggregated to the learnt enhanced answer, each of the nodes for the observed answers being connected to at least one node representing a learnt expertise indicator of a judge who gave the observed answer.

19. A method as claimed in claim 18 wherein using the probabilistic learning system to determine enhanced answers to the questions comprises propagating answers made by the judges through nodes in a logical component of the probabilistic learning system according to logical relations between the questions.