Computer system and computer-implemented process for analyzing results of cytology tests for performance evaluation of cytologists

• US 6,468,208 B1
• Filed: 05/26/1999
• Issued: 10/22/2002
• Est. Priority Date: 05/29/1998
• Status: Expired due to Fees

• Current Assignee: Brigham and Women's Hospital Incorporated
• Original Assignee: Brigham and Women's Hospital Incorporated
• Inventors: Cibas, Edmund S.
• Primary Examiner(s): Nasser, Robert L.
• Assistant Examiner(s): MALLARI, PATRICIA C
• Application Number:
  US09/318,871
• Time in Patent Office: 1,245 Days
• Field of Search: 600/300, 600/301, 128/920-925
• US Class Current: 600/300
• CPC Class Codes: G16H 10/40 for data related to laboratory analysis, e.g. patient specimen analysis
• CPC Class Codes: G16H 40/20 for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
• CPC Class Codes: Y10S 128/92 Computer assisted medical diagnostics
• CPC Class Codes: Y10S 128/923 by comparison of patient data to other data

Abstract

The evaluation of the performance of cytotechnologists and cytopathologists is obtained through statistical analyses of a computer system database including data representative of diagnosis by the cytotechnologists and cytopathologists. The statistical analyses provide objective measures of performance that aid in improving the quality of evaluation in the cytology laboratory. The locator skills, interpretive skills, volume statistics and productivity of the cytotechnologists is evaluated based on false-negative fraction information, divergent percentage calculations, average scores and kappa values, z scores and pro-rata workload measurements. The accuracy and performance of the cytopathologists is evaluated through the use of receiver operating characteristic curves and positive predictive values.

Claims

1. A method for evaluating the performance of an individual in a cytology laboratory comprising steps of:

collecting a plurality of diagnoses from at least one individual;

comparing the plurality of diagnoses from the at least one individual against a plurality of expected diagnoses to identify divergent diagnoses;

statistically quantifying the divergence between the plurality of diagnoses and the plurality of expected diagnoses, wherein statistically quantifying the divergence includes the step of generating statistical evaluation information quantifying locator skills and interpretive skills of the individual; and

graphically displaying the statistical quantification of the divergence.

2. The method according to claim 1, wherein the evaluation information further quantifies productivity and volume statistics of the individual.

3. The method according to claim 1, wherein the step of statistically quantifying the divergence includes the step of graphically displaying a true-positive diagnosis rate against a false-negative diagnosis rate to provide a receiver operating characteristic (ROC) curve for the at least one individual.

4. A method for evaluating the performance of an individual in a cytology laboratory comprising steps of:

collecting a plurality of diagnoses from at least one individual;

comparing the plurality of diagnoses from the at least one individual against a plurality of expected diagnoses to identify divergent diagnoses;

statistically quantifying the divergence between the plurality of diagnoses and the plurality of expected diagnoses, wherein statistically quantifying the divergence includes the step of generating a positive predictive value (PPV) for the individual using the below equation;

$\mathrm{PPV}=\frac{\mathrm{Number}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{True}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{Positives}}{\mathrm{Number}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{True}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{Positives}+\mathrm{Number}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{False}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{Positives}}$

wherein the Number True Positives corresponds to a number of the plurality of diagnoses that indicate a positive diagnosis and match a corresponding one of the plurality of expected diagnoses, and wherein the Number False Positives corresponds to the number of the plurality of diagnoses that indicate a positive diagnosis and do not match a corresponding one of the plurality of expected diagnoses; and

graphically displaying the statistical quantification of the divergence.

5. The method according to claim 4, wherein the step of statistically quantifying the divergence includes the step of graphically displaying a true-positive diagnosis rate against a false-negative diagnosis rate to provide a receiver operating characteristic (ROC) curve for the at least one individual.

6. A method for evaluating the performance of a technologist comprising the steps of:

collecting provisional diagnosis data from at least one technologist for a plurality of cases;

selecting a set of the plurality of cases for re-screening by at least one cytologist, the re-screening providing final diagnosis data; and

comparing the final diagnosis data to the provisional diagnosis data to generate statistical evaluation information quantifying locator skills and interpretive skills of the at least one technologist.

7. The method according to claim 6, wherein the statistical evaluation information further includes statistical information quantifying the productivity and volume of the at least one technologist.

8. The method according to claim 6, wherein the information quantifying the locator skills of the at least one technologist includes false-negative fraction information indicating a number of cases in the set of cases selected for re-screening having a provisional diagnosis differing from the final diagnosis, and wherein the step of comparing further comprises the step of generating a false-negative fraction to represent the performance of the at least one technologist.

9. The method according to claim 8, wherein the set of the plurality of cases comprises a first number of cases of a first type and a second number of cases of a second type, and wherein the step of generating a false-negative fraction (FNF1) is determined according to the below equation:

$\mathrm{FNF1}=\frac{\mathrm{hqcr}+(\mathrm{rqcr}*\left(\left(\mathrm{totn}-\mathrm{hqc}\right)/\mathrm{rqc}\right)}{\left(\mathrm{eca}+\mathrm{uns}+\mathrm{hqcr}\right)+\left(\mathrm{rqcr}*\left(\left(\mathrm{totn}-\mathrm{hqc}\right)/\mathrm{rqc}\right)\right)}$

wherein;

hqc=the first number of the first type of cases

rqc=the second number of the second type of cases

hqcr=a number of the first type of cases wherein the provisional diagnosis differs from the final diagnosis

rqcr=a number of the second type of cases wherein the provisional diagnosis differs from the final diagnosis

haty=a number of the first type of cases having a final diagnosis of atypical

raty=a number of the second type of cases having a final diagnosis of atypical

taty=a total number of the set of cases having a provisional diagnosis of atypical

eca=a total number of the set of cases having a final diagnosis of abnormal

uns=a total number of the set of cases having a final diagnosis of unsatisfactory

totn=a total number of the set of cases having a final diagnosis of normal.

10. The method according to claim 8. wherein the set of the plurality of cases comprises a first number of cases of a first type and a second number of cases of a second type, and wherein the step of generating the false-negative fraction (FNF2) uses the below equation:

$\mathrm{FNF2}=\frac{\left(\mathrm{hqcr}-\mathrm{haty}\right)+\left(\left(\mathrm{rqcr}-\mathrm{raty}\right)*\left(\left(\mathrm{totn}-\mathrm{hqc}\right)/\mathrm{rqc}\right)\right)}{\left(\mathrm{eca}-\mathrm{taty}\right)+\left(\mathrm{uns}\right)+\left(\mathrm{hqcr}-\mathrm{haty}\right)+\left(\left(\mathrm{rqcr}-\mathrm{raty}\right)*\left(\left(\mathrm{totn}-\mathrm{hqc}\right)/\mathrm{rqc}\right)\right)}$

hqc=the first number of the first type of cases

rqc=the second number of the second type of cases

hqcr=a number of the first type of cases wherein the provisional diagnosis differs from the final diagnosis

rqcr=a number of the second type of cases wherein the provisional diagnosis differs from the final diagnosis

haty=a number of the first type of cases having a final diagnosis of atypical

raty=a number of the second type of cases having a final diagnosis of atypical

taty=a total number of the set of cases having a provisional diagnosis of atypical

eca=a total number of the set of cases having a final diagnosis of abnormal

uns=a total number of the set of cases having a final diagnosis of unsatisfactory

totn=a total number of the set of cases having a final diagnosis of normal.

11. The method according to claim 6, wherein the information quantifying the interpretive skills of the technologist includes a discordant percentage, and wherein the step of comparing further includes a step of generating the discordant percentage using the below equation:

$\%<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{Discordant}=\frac{\mathrm{Number}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{discordant}}{\mathrm{Number}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{of}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{the}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{plurality}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{of}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{cases}}$

12. The method according to claim 6, wherein the information quantifying the interpretive skills of the technologist includes an average score, and wherein the step of comparing further includes a step of generating the average store, the step of generating including the steps of:

storing, in an array comprising a plurality of entries arranged in a plurality of rows and a corresponding plurality of columns, each row and column associated with one of a plurality of types of diagnoses, a number of each of the provisional and final diagnoses of the types indicated by the row and column;

applying a weighted value to each of the plurality of entries to provide a plurality of weighted entries, the weighted value corresponding to a degree of error between the provisional and final diagnoses associated with the types indicated by the row and column; and

summing the weighted entries to provide the average score.

13. The method according to claim 6, wherein the information quantifying the interpretive skills of the technologist includes a kappa (κ

) value, and wherein the step of comparing further includes a step of generating the kappa value, the step of generating including the steps of;

storing, in an array comprising a plurality of entries arranged in a plurality of rows and a corresponding plurality of columns, each row and column associated with one of a plurality of types of diagnoses, a frequency of the provisional and final diagnoses of the types indicated by the row and column; and

generating the κ

value according to the below equation, where n corresponds to a number of the plurality of types of diagnoses;

$\mathrm{\kappa <br><br>}=\frac{\mathrm{Po}-\mathrm{Pe}}{1-\mathrm{Pe}}$

where;

$\mathrm{Po}=\underset{i=0,j=0}{\overset{i=n,j=n}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{Array}<br><br>\left(i,j\right)$$\mathrm{Pe}=\underset{i=0}{\overset{i=n}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(\sum <br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{Array}<br><br>\text{\hspace{1em}<br><br>}<br><br>\left({\mathrm{Row}}_i\right)\times <br><br>\sum <br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{Array}<br><br>\text{\hspace{1em}<br><br>}<br><br>\left({\mathrm{Column}}_i\right)\right).$

14. The method according to claim 6, wherein the information quantifying the volume statistics of the technologist includes a first percentage of the plurality of diagnoses having an unsatisfactory diagnosis and a second percentage of the plurality of diagnoses having an abnormal diagnosis.

15. The method according to claim 14, wherein the information quantifying the volume statistics of the technologist includes a first z score, relating the first percentage of the technologist to a percentage of a first type of diagnoses in the laboratory, and a second z score relating the second percentage to a percentage of a second type of diagnoses in the laboratory, and wherein the respective first and second z scores are determined according to the below equation:

$z=\frac{x-\mathrm{\mu <br><br>}}{\mathrm{\sigma <br><br>}}$

where x is a value of respective first or second percentage, μ

is a mean of the distribution of the diagnoses of the respective first or second type, and σ

is a standard deviation of the distribution of the diagnoses of the respective first or second type.

16. A computer system for evaluating performance of technologists including a processor capable of performing the steps of:

collecting provisional diagnosis data from at least one technologist for a plurality of cases;

selecting a set of the plurality of cases for re-screening, the re-screening providing final diagnosis data; and

comparing the final diagnosis data to the provisional diagnosis data to generate statistical evaluation information quantifying locator skills and interpretive skills of the at least one technologist.

17. The computer system according to claim 16, wherein the information quantifying the locator skills of the at least one technologist includes false-negative fraction information indicating a number of cases in the set of cases selected for re-screening having a provisional diagnosis differing from the final diagnosis.

18. The computer system according to claim 17, wherein the set of the plurality of cases comprises a first number of cases of a first type and a second number of cases of a second type, and wherein the false-negative fraction (FNF2) information is determined according to the below equation:

$\mathrm{FNF2}=\frac{\left(\mathrm{hqcr}-\mathrm{haty}\right)+\left(\left(\mathrm{rqcr}-\mathrm{raty}\right)*\left(\left(\mathrm{totn}-\mathrm{hqc}\right)/\mathrm{rqc}\right)\right)}{\left(\mathrm{eca}-\mathrm{taty}\right)+\left(\mathrm{uns}\right)+\left(\mathrm{hqcr}-\mathrm{haty}\right)+<br><br>\left(\left(\mathrm{rqcr}-\mathrm{raty}\right)*\left(\left(\mathrm{totn}-\mathrm{hqc}\right)/\mathrm{rqc}\right)\right)}$

hqc=the first number of the first type of cases

rqc=the second number of the second type of cases

hqcr=a number of the first type of cases wherein the provisional diagnosis differs from the final diagnosis

rqcr=a number of the second type of cases wherein the provisional diagnosis differs from the final diagnosis

haty=a number of the first type of cases having a final diagnosis of atypical

raty=a number of the second type of cases having a final diagnosis of atypical

taty=a total number of the set of cases having a provisional diagnosis of atypical

eca=a total number of the set of cases having a final diagnosis of abnormal

uns=a total number of the set of cases having a final diagnosis of unsatisfactory

totn=a total number of the set of cases having a final diagnosis of normal.

19. The computer system according to claim 17, wherein the set of the plurality of cases comprises a first number of cases of a first type and a second number of cases of a second type, and wherein the false-negative fraction (FNF1) information is determined according to the below equation:

$\mathrm{FNF1}=\frac{\mathrm{hqcr}+(\mathrm{rqcr}*\left(\left(\mathrm{totn}-\mathrm{hqc}\right)/\mathrm{rqc}\right)}{\left(\mathrm{eca}+\mathrm{uns}+\mathrm{hqcr}\right)+\left(\mathrm{rqcr}*\left(\left(\mathrm{totn}-\mathrm{hqc}\right)/\mathrm{rqc}\right)\right)}$

wherein;

hqc=the first number of the first type of cases

rqc=the second number of the second type of cases

hqcr=a number of the first type of cases wherein the provisional diagnosis differs from the final diagnosis

rqcr=a number of the second type of cases wherein the provisional diagnosis differs from the final diagnosis

haty=a number of the first type of cases having a final diagnosis of atypical

raty=a number of the second type of cases having a final diagnosis of atypical

taty=a total number of the set of cases having a provisional diagnosis of atypical

eca=a total number of the set of cases having a final diagnosis of abnormal

uns=a total number of the set of cases having a final diagnosis of unsatisfactory

totn=a total number of the set of cases having a final diagnosis of normal.

20. The computer system according to claim 16, wherein the statistical evaluation information further includes statistical information quantifying the productivity and volume of the at least one technologist.

21. The computer system according to claim 20 further comprising a graphical user interface for displaying the statistical evaluation information for the at least one technologist.

22. A computer system for evaluating the accuracy of cytological diagnoses comprises:

means for periodically receiving and storing evaluation data for a plurality of cases of cytology slides, the evaluation data including, for each of the plurality of cases, an identifier of an individual that evaluated the case, at least one diagnosis of at least one slide associated with the case, and a time stamp indicating when the respective case was diagnosed;

means for generating statistical evaluation information quantifying the locator skills and interpretive skills of each of a plurality of individuals, using the evaluation data.