System and method for consensus-based knowledge validation, analysis and collaboration

• US 7,970,592 B2
• Filed: 10/29/2009
• Issued: 06/28/2011
• Est. Priority Date: 09/01/2004
• Status: Active Grant

• Current Assignee: Nytell Software LLC (Intellectual Ventures LLC)
• Original Assignee: TTI Inventions C LLC (Intellectual Ventures LLC)
• Inventors: Behrens, Clifford A., Shim, Hyong-sop
• Primary Examiner(s): PHAN, THAI Q
• Application Number:
  US12/608,537
• Publication Number:
  US 20100305915A1
• Time in Patent Office: 607 Days
• Field of Search: 703/2, 702/179, 702/183, 702/188, 702/186
• US Class Current: 703/2
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G09B 5/00 Electrically-operated educational appliances working with questions and answers G09B7/00; simulators G09B9/00; advertising or displaying in general G09F

Abstract

A consensus-based knowledge validation and analysis system provides a way to increase use of collaboration tools among panels of experts by providing a system for analyzing and validating the responses of such experts to a set of questions. The system uses a set of response data input by a panel of experts with respect to a particular subject matter formatted in accordance with a data model as input. The response data set is used to estimate an empirical point estimate matrix indicative of the amount of agreement in the responses on all items between the panelists. The empirical point estimate matrix is used to estimate the saliency of the subject matter to panelists, the competency of each panelist and a consensus model of correct answers is based on the estimated competency of each panelist and the of responses for each item in the response data set. This consensus model is used to generate a knowledge map to aid visualization of the consensus data and encourage further collaboration and consensus building. The method is implemented in a web-based system that enables users of collaboration tools to send response data sets to the tool via the Internet or virtual private network and to likewise retrieve knowledge maps, panelist information and consensus data. An interactive feature enables users/panelists to collaborate with other panelists using the knowledge map as an interface to one or more collaboration tools such as instant messaging.

Claims

1. A tangible computer-readable medium having instructions for comparing a plurality of knowledge maps derived from a consensus model generated from a set of response data collected from a plurality of panelists using a collaborative modeling tool over a period of time stored thereon, the instructions configured to cause a computing device to:

input a first set of response data from the plurality of panelists using a collaborative modeling tool at a first time;

compute a first consensus model from the first set of response data;

develop a first knowledge map from the first consensus model;

input a second set of response data from the plurality of panelists using the collaborative modeling tool at a second time, wherein the second time is different than the first time;

compute a second consensus model from the second set of response data;

develop a second knowledge map from the second consensus model;

perform a procrustes analysis of the second knowledge map against the first knowledge map to generate a realigned second knowledge map;

enable display of the first knowledge map and the realigned second knowledge map.

2. The tangible computer readable medium of claim 1 the instructions further comprising instructions configured to cause a computing device to calculate a compactness metric indicative of the an overall knowledge variability between the plurality of panelists.

3. The tangible computer readable medium of claim 2 wherein the instructions configured to cause a computing device to perform the procrustes analysis further comprises instructions configured to cause a computing device to:

rotate the coordinates in the second knowledge map;

calculate a residual sum of the squared differences between the coordinates in Euclidean multivariate space; and

derive a correlation-like statistic from the residual sum until the correlation-like statistic approaches one.

4. The tangible computer-readable medium of claim 1 the instructions further comprising instructions configured to cause a computing device to:

enable display of a scatter plot of the first and second knowledge maps; and

enable display of a plot showing movement of the plurality of panelists between the first knowledge map and second knowledge map.

5. The tangible computer-readable medium of claim 1 the instructions further comprising instructions configured to cause a computing device to:

enabling the plurality of panelists to communicate by selecting an identifier in the knowledge map, scatter plot, or plot showing movement of the plurality of panelists where the identifier is associated with one of the plurality of panelists.

6. The tangible computer-readable medium of claim 5 wherein the instructions configured to cause a computing device to enable the plurality of panelists to communicate further comprises instructions configured to cause a computing device to present the plurality of panelists with a graphical user interface to one or more collaboration tools when an identifier is selected.

7. A tangible computer-readable medium having instructions stored thereon, the instructions configured to cause a computing device to:

derive a knowledge map from a set of response data collected from a plurality of panelists in response to a plurality of items regarding a subject matter, the knowledge map comprising a graphical display of a comparison of responses of the plurality of panelists;

present the knowledge map to one or more of the plurality of panelists;

enable a first panelist from the plurality of panelists to initiate communication with a second panelist from the plurality of panelists by facilitating a selection of an identifier associated with the second panelist on the knowledge map.

8. The tangible computer-readable medium of claim 7 wherein the instructions configured to cause a computing device to enable further comprises instructions configured to cause a computing device to present the plurality of panelists with a graphical user interface to one or more collaboration tools after the identifier is selected.

9. The tangible computer-readable medium of claim 8 wherein the collaboration tool is an instant messaging system.

10. The tangible computer readable medium of claim 7 wherein the plurality of panelists are at least compared in terms of estimated competencies and relative differences in the domain knowledge of the plurality of panelists.