Method and apparatus for evaluating data and implementing training based on the evaluation of the data

• US 6,795,793 B2
• Filed: 07/19/2002
• Issued: 09/21/2004
• Est. Priority Date: 07/19/2002
• Status: Expired due to Fees

• Current Assignee: Med-Ed Innovations Incorporated
• Original Assignee: Med-Ed Innovations Incorporated
• Inventors: Sudderth, Erik B., Stahl, Stephen M., Shayegan, Darius K., McElroy, Tucker S.
• Primary Examiner(s): Hoff, Marc S.
• Assistant Examiner(s): SUAREZ, FELIX E
• Application Number:
  US10/199,969
• Publication Number:
  US 20040015329A1
• Time in Patent Office: 795 Days
• Field of Search: 702/179, 702/186, 702/188, 702/198, 702/128, 702/176, 702/177, 702/181, 702/182, 379/15.02, 705/8, 705/7, 705/10, 717/106
• US Class Current: 702/179
• CPC Class Codes: G09B 23/28 for medicine

Abstract

The present invention provides systems and methods for statistically analyzing collected data and providing training for the collection of the data. In one embodiment, the system and method are configured to collect data about a plurality of objects and determine a measure of agreement between the collected data and a benchmark. Based on the measure of agreement reliability and a variance of the collected data are determined relative to the benchmark. In training a plurality of raters are instructed to rate at least one of a plurality of objects such that each rater collects data that rates at least one of the objects. A measure of agreement of the data collected by a rater is determined relative to the benchmark, and reliability and variance of the data collected are determined, which are used to evaluate the rater.

Claims

1. A method for analyzing collected data, comprising:

collecting data about a plurality of objects;

determining a measure of agreement between the collected data and a benchmark;

determining a reliability of the collected data;

determining a variance of the collected data relative to the benchmark; and

performing a significance assessment based on the variance of the collected data;

wherein the step of performing the significance assessment includes assessing the collected data relative to a random data distribution.

2. A method for analyzing collected data, comprising:

collecting data about a plurality of objects;

determining a measure of agreement between the collected data and a benchmark;

determining a reliability of the collected data;

determining a variance of the collected data relative to the benchmark;

determining an erraticity of each of the plurality of objects; and

weighting the collected data based on the determined erraticity of at least one of the plurality of objects.

3. The method as claimed in claim 2, further comprising the steps of:

determining weighted mean squared errors (MSE) or the collected data based on the determined erraticity; and

the step of weighting the collected data including applying the weighted MSE.

4. A method for training a plurality of raters for rating at least one of a plurality of objects, the method comprising:

instructing a plurality of raters regarding performing a rating of at least one of a plurality of objects;

allowing each of the plurality of raters to collect data that rates at least one of the plurality of objects;

determining a measure of agreement of the data collected by a single rater with benchmark data;

determining the reliability of the data collected by the single rater;

determining the variance of the data collected by the single rater relative to the benchmark data;

evaluating the single rater relative to the benchmark data;

determining an erraticity for each of the plurality of objects;

weighting the collected data of an object of the plurality of objects based on the erraticity of the object; and

the step of determining the variance including determining the variance based on the weighted data for the single rater.

5. A method of statistically analyzing data, comprising:

collecting a plurality of sets of data based from a plurality of evaluations of an object, where each set of data includes a plurality of data;

comparing each set of data with each of the other sets of data;

determining a deviation of the sets of data;

determining a difficulty in evaluating the object base on the determination of the deviation of the sets of data;

wherein the collecting the plurality of sets of data includes collecting the plurality of sets of data based on a plurality of expert evaluations of the object, such that the plurality of sets of data are a plurality of expert sets of data;

collecting a plurality of rating sets of data from a plurality of raters evaluating the object;

determining a measure of disagreement between each rating set of data and each expert set of data;

establishing a weight-correction of each measure of disagreement based on the difficulty in evaluating the object;

generating a weight corrected measure of disagreement;

determining if each weight corrected measure of disagreement exceeds a threshold; and

retraining a rater who generated the rating set of data having weight corrected measure of disagreement that exceeds the threshold.

6. A method of statistically analyzing data, comprising:

collecting a plurality of sets of data based from a plurality of evaluations of an object, where each set of data includes a plurality of data;

comparing each set of data with each of the other sets of data;

determining a deviation of the sets of data; and

determining a difficulty in evaluating the object based on the determination of the deviation of the sets of data;

wherein the establishing the weight-correction for each measure of disagreement includes a maximum disagreement weight.

7. A system for statistically analyzing data, comprising:

means for collecting a set of data from a rater;

means for comparing the set of data with a standards of convention;

means for determining a distance measure between the set of data and the standards of convention;

means for evaluating the set of data based on the distance measure;

means for certifying the data if the evaluation exceeds a threshold;

means for establishing a null hypotheses;

means for determining if the set of data exceeds a threshold; and

means for rejecting the null hypotheses if the distance measure exceeds the threshold.

8. A system for statistically analyzing data, comprising:

means for collecting a set of data from a rater;

means for comparing the set of data with a standards of convention;

means for determining a distance measure between the set of data and the standards of convention;

means for evaluating the set of data based on the distance measure;

means for certifying the data if the evaluation exceeds a threshold;

means for establishing a null distribution;

means for establishing a threshold based on the null distribution; and

means for determining if the set of data exceeds the threshold, such that the means for certifying certifies the set of data if the set of data exceeds the threshold.