System and method for consensus-based knowledge validation, analysis and collaboration

• US 20070239405A1
• Filed: 09/01/2005
• Published: 10/11/2007
• Est. Priority Date: 09/01/2004
• Status: Active Grant

• Current Assignee: Nytell Software LLC (Intellectual Ventures LLC)
• Original Assignee: Telcordia Licensing Co LLC (Telefonaktiebolaget LM Ericsson)
• Inventors: Shim, Hyong-sop, Behrens, Clifford
• Granted Patent: US 7,630,867 B2
• Time in Patent Office: Days
• US Class Current: 703/2
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G09B 5/00 Electrically-operated educational appliances working with questions and answers G09B7/00; simulators G09B9/00; advertising or displaying in general G09F

Abstract

A consensus-based knowledge validation and analysis system provides a way to increase use of collaboration tools among panels of experts by providing a system for analyzing and validating the responses of such experts to a set of questions. The system uses a set of response data input by a panel of experts with respect to a particular subject matter formatted in accordance with a data model as input. The response data set is used to estimate an empirical point estimate matrix indicative of the amount of agreement in the responses on all items between the panelists. The empirical point estimate matrix is used to estimate the saliency of the subject matter to panelists, the competency of each panelist and a consensus model of correct answers is based on the estimated competency of each panelist and the of responses for each item in the response data set. This consensus model is used to generate a knowledge map to aid visualization of the consensus data and encourage further collaboration and consensus building. The method is implemented in a web-based system that enables users of collaboration tools to send response data sets to the tool via the Internet or virtual private network and to likewise retrieve knowledge maps, panelist information and consensus data. An interactive feature enables users/panelists to collaborate with other panelists using the knowledge map as an interface to one or more collaboration tools such as instant messaging.

Claims

1. A method for consensus-based validation and analysis of response data gathered from a plurality of panelists in response to a plurality of items regarding a subject matter comprising the steps of:

collecting a set of response data from each of the plurality of panelists for each of the plurality of items;

formatting the response data from each of the plurality of panelists into a response data matrix in accordance with a predetermined data model;

estimating from the response data matrix the empirical point estimate matrix indicative of the amount of agreement in the responses on all items between panelists;

determining the saliency of the subject matter to the plurality of panelists;

estimating the competency of each panelist based on the empirical point estimate matrix; and

,

computing the consensus model of correct answers based on the estimated competency of each panelist and the responses for each item in the response data set.

2. The method of claim 1 further comprising the step of developing a knowledge map from the consensus model.

3. The method of claim 2 further comprising the step of displaying the knowledge map and information about the panelists to a user.

4. The method of claim 1 wherein the step of collecting a set of response data from each of a plurality of panelists is performed using a collaborative modeling tool.

6. The method of claim 1 wherein the step of estimating from the response data matrix the empirical point estimate matrix, M*_ij, indicative of the amount of agreement in the responses on all items between panelists are concordance correlation coefficients given by:

7. The method of claim 1 wherein the step of estimating the competency of each panelist based on the empirical point estimate matrix results in matrix D_i.

8. The method of claim 1 wherein the step of computing the consensus model of correct answers based on the estimated competency of each panelist and the of responses for each item in the response data set is performed by calculating the weighted average of the response to item k by panelist i, and, D*_i, is the estimate of competence of panelist i in accordance with the following algorithm:

9. The method of claim 2 further comprising the step of enabling the user to collaborate with a panelist using the knowledge map as an interface to one or more collaboration tools.

5. The method of claim 5 wherein the collaborative modeling tool is the SIAM collaboration tool.

10. A method for consensus-based validation and analysis of one or more sets of response data gathered from a plurality of panelists in response to a plurality of items regarding a subject matter using a collaborative modeling tool comprising the steps of:

inputting the response data from the collaborative modeling tool formatted into a response data matrix in accordance with a predetermined data model;

estimating from the response data matrix the empirical point estimate matrix indicative of the amount of agreement in the responses on all items between panelists;

determining the saliency of the subject matter to the plurality of panelists;

estimating the competency of each panelist based on the empirical point estimate matrix; and

,

computing the consensus model of correct answers based on the estimated competency of each panelist and the responses for each item in the response data set.

11. The method of claim 10 further comprising the step of developing a knowledge map from the consensus model.

12. The method of claim 11 further comprising the step of displaying the knowledge map and information about the panelists to a user.

13. The method of claim 11 further comprising the step of providing the user with an interface to one or more of the plurality of panelists through the knowledge map.

14. The method of claim 13 wherein the step of providing further comprises the step of enabling the user to access one or more collaboration tools through the knowledge map.

15. The method of claim 14 wherein the step of enabling is accomplished by launching an instant messaging session.

16. A system for the validation and analysis of knowledge from a set of response data collected from a plurality of panelists regarding a subject matter using a collaborative modeling tool:

means for formatting the response data from the collaborative modeling tool into a response data matrix in accordance with a predetermined data model;

means for estimating from the response data matrix the empirical point estimate matrix indicative of the amount of agreement in the responses on all items between panelists;

means for determining the saliency of the subject matter to the plurality of panelists;

means for estimating the competency of each panelist based on the empirical point estimate matrix; and

,

means for computing the consensus model of correct answers based on the estimated competency of each panelist and the responses for each item in the response data set.

17. The system of claim 16 further comprising the means for developing a knowledge map from the consensus model.

18. The system of claim 17 further comprising the step of displaying the knowledge map and information about the panelists to a user

19. The system of claim 16 further comprising means estimating the empirical point estimate matrix, the means for estimating the competency of each panelist and the means for computing the consensus model are implemented in software developed in accordance with the R programming language.

20. A method for a user to perform the comparison of a plurality of knowledge maps each derived from a consensus model generated from a set of response data collected from a plurality of panelists using a collaborative modeling tool over a period of time comprising the steps of:

inputting a first set of response data from the collaborative modeling tool;

computing a first consensus model from the first set of response data;

developing a first knowledge map from the first consensus model;

inputting a second set of response data from the same set of panelists at a later point in time;

computing a second consensus model from the second set of response data;

developing a second knowledge map from the second consensus model;

performing a procrustes analysis of the second knowledge map against the first knowledge map to generate a realigned second knowledge map;

presenting the first knowledge map and second realigned knowledge maps to the user.

21. The method of claim 20 wherein the method further comprises the step of calculating a compactness metric indicative of the overall knowledge variability between panelists.

22. The method of claim 21 wherein the step of performing the procrustes analysis further comprises iteratively performing the steps of:

rotating the coordinates in the second knowledge map;

calculating a residual sum of the squared differences between coordinates in Euclidean multivariate space; and

,

deriving a correlation-like statistic from the residual sum until the correlation-like statistic approaches one.

23. The method of claim 20 further comprising the steps of:

displaying to the user a scatter plot of the original first and second knowledge maps;

displaying to the user a plot showing the movement of each of the plurality of panelists between the first knowledge map and second knowledge map.

24. The method of claim 20 further comprising the steps of:

enabling a first panelist to communicate with a second panelist by selecting the identifier in the knowledge map, scatter plot or plot showing movement of panelists where such identifier is associated with the second panelist on the knowledge map.

25. The method of claim 24 wherein the step of enabling further comprises the step of presenting the first panelist with a graphical user interface to one or more collaboration tools when the first panelists selects an identifier.

26. A method of knowledge-based collaboration comprising the steps of:

deriving a knowledge map from a set of response data collected from a plurality of panelists in response to a plurality if items regarding a subject matter;

presenting the knowledge map to one or more of the plurality of panelists;

enabling a first panelist to communicate with a second panelist by selecting the identifier associated with the second panelist on the knowledge map.

27. The method of claim 26 wherein the step of enabling further comprises the step of presenting the first panelist with a graphical user interface to one or more collaboration tools when the first panelists selects an identifier.

28. The method of claim 27 wherein the collaboration tool is an instant messaging system.