Inter-laboratory performance monitoring system

• US 5,532,941 A
• Filed: 07/08/1994
• Issued: 07/02/1996
• Est. Priority Date: 07/08/1994
• Status: Expired due to Fees

• Current Assignee: Medical Analysis Systems, Inc. (Thermo Fisher Scientific Incorporated)
• Original Assignee: Dade International, Inc. (Siemens AG)
• Inventors: Lin, Lawrence I.
• Primary Examiner(s): Voeltz, Emanuel T.
• Assistant Examiner(s): Shah, Kamini S.
• Application Number:
  US08/272,239
• Time in Patent Office: 725 Days
• Field of Search: 364/552, 364/554, 364/550, 364/571.01-571.08, 364/582, 364/138, 364/468
• US Class Current: 702/84
• CPC Class Codes: G01N 2035/00653 statistical methods comparing labs or apparatuses
• CPC Class Codes: G01N 35/00594 Quality control, including calibration or testing of components of the analyser
• CPC Class Codes: G01N 35/00603 Reinspection of samples
• CPC Class Codes: G01N 35/00712 Automatic status testing, e.g. at start-up or periodic
• CPC Class Codes: G01N 35/00871 Communications between instruments or with remote terminals
• CPC Class Codes: G05B 19/41875 characterised by quality surveillance of production
• CPC Class Codes: G05B 2219/31392 Lims laboratory information and management system
• CPC Class Codes: G05B 2219/32187 Correlation between controlling parameters for influence on quality parameters
• CPC Class Codes: G08C 25/00 Arrangements for preventing or correcting errors; Monitoring arrangements
• CPC Class Codes: G16H 10/40 for data related to laboratory analysis, e.g. patient specimen analysis
• CPC Class Codes: G16H 40/40 for the management of medical equipment or devices, e.g. scheduling maintenance or upgrades
• CPC Class Codes: Y02P 90/02 Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Abstract

A system and method for producing quality control evaluation information for each instrument in a large group of instruments making up a peer group which periodically (such as daily) run a set of control samples from a common lot of control materials. The control data from all instruments is reported electronically to a central station which collects all of the data and stores it along with identification information. The central station selects a golden peer group of the instruments which meet certain minimum operating criteria, and determines control targets from the golden peer group using techniques which assure horizontal and vertical robustness, so as to eliminate the need for any manual editing of the control data. A concordance correlation coefficient is determined for each instrument against the control targets, with the CCC covering an interval (such as a month). The distribution of CCC'"'"'s for all instruments in the peer group is determined, and each instrument is rated with respect to the distribution. The CCC evaluation is also used to select the golden peer group for the next interval.

Claims

1. A system for quality control monitoring the operating performance of a large plurality of instruments, the system comprising, in combination:

a plurality of geographically distributed instruments comprising a peer group operated to produce control data on a periodic basis and also experimental data whose quality is to be monitored;

a communication medium linking the instruments to a central station for reporting the control data to the central station;

the central station including a communication interface linked to the communication medium, a data storage module for the control data, a target module for setting control targets, a status memory, a correlation comparator for comparing the control data against the targets, and a CCC performance evaluator;

the data storage module comprising means for storing control data in conjunction with instrument identification data received on a periodic basis via the communication interface from the instruments making up the peer group;

the status memory containing indicators identifying a subset of instruments within the peer group which are to serve as the golden peer group;

the target module including means responsive to the status memory for selecting control data from the golden peer group, automatically processing the selected control data horizontally across periods and vertically across instruments to minimize the effect of outliers, and to produce from the so-processed control data a set of periodic moving targets for the control data;

the correlation comparator comprising means for comparing the periodic targets against the control data for each instrument to determine for each instrument a CCC for an interval comprising a number of periods which rates the accuracy and precision of the instrument performance for said interval with respect to the moving targets for the same interval;

the CCC evaluator comprising means for determining the distribution of all of the instruments in the peer group and rating the CCC for each instrument against said distribution; and

means responsive to the CCC evaluator for selecting a subset of good performing instruments near the center of the distribution of the peer group as comprising the golden peer group and operative on the status memory for updating the golden peer group identification.

2. The system according to claim 1 in which the distribution of CCC'"'"'s among all instruments in the peer group comprises benchmark average and standard deviations of CCC'"'"'s among all instruments in the peer group, and the means responsive to the CCC evaluator selects all instruments within a predetermined standard deviation of the average as the good performing laboratories comprising the golden peer group.

3. The system according to claim 2 wherein the predetermined standard deviation is about 1.

4. The system according to claim 1 wherein the control data produced on a periodic basis by each instrument comprises at least high and low data points spanning a desired operating range, and the set of periodic moving targets comprises at least high and low targets for each period.

5. The system according to claim 4 in which the periodic basis on which the control data is received is daily, and the interval over which the correlation comparator operates is on the order of monthly.

6. The system according to claim 5 in which the correlation comparator operates over dual intervals, one being a calendar month, and a second being the most recent d days comprising a "current" interval.

7. The system according to claim 6 in which the CCC evaluator updates the current interval daily, and the means responsive to the CCC evaluator updates the status memory over an interval substantially longer than daily.

8. The system according to claim 7 in which the means responsive to the CCC evaluator updates the status memory on about a monthly basis.

9. The system according to claim 7 in which the rating produced by the CCC evaluator for each instrument is reported to the instrument, such report being made within about a day or less of the receipt of the periodic control data.

10. The system according to claim 1 in which the target module further includes means for accumulating a set of fixed targets for the control data comprising an average of the periodic moving targets;

the correlation comparator also comprising means for comparing the fixed targets against the periodic control data for each instrument to determine for each laboratory a CCC for the interval with respect to the fixed targets; and

the CCC evaluator also comprising means for rating the fixed target CCC for each instrument against the distribution of CCC'"'"'s among all instruments in the peer group.

11. The system according to claim 1 in which the horizontal processing across periods includes a moving median memory for storing n most recent periods of control data for each instrument, the moving median memory being operated to (a) store new control data for a new period, (b) delete the control data from the oldest period, and (c) determine and output the median for the new n periods and as the control data for that instrument for the new period.

12. The system according to claim 11 in which the vertical processing across instruments comprises a two-step M-estimation process for producing an average value from all instruments in the golden peer group for each period, the M-estimation process being performed on the periodic control data output from the moving median memory for each instrument in the golden peer group.

13. The system according to claim 1 in which a CCC memory stores the CCC of each instrument for each interval, the CCC evaluator comprising means for determining benchmark average and standard deviations of CCC'"'"'s among all instruments in the peer group for each interval, the means for rating comprising means for comparing the CCC for each instrument with the average and standard deviations determined from all instruments in the peer group.

14. The system according to claim 1 wherein the CCC for each instrument is determined against targets taken from only the golden peer group, but the CCC for each laboratory is evaluated against the CCC'"'"'s taken from all laboratories in the peer group, thereby to enhance the precision of the CCC while ranking the determined CCC against the entire peer group.

15. A system for providing quality control ratings for a plurality of instruments which are operated on a periodic basis to produce control data from the same lot of control materials for all instruments comprising a peer group, the system comprising, in combination:

an electronic communication interface for automatically receiving from all instruments in the peer group the control data on a periodic basis;

a data storage module for storing the control data received on a periodic basis from all of the instruments in the peer group, the data storage module storing said control data along with an identification of the instrument and an identification of the period for which the control data was produced;

a status memory identifying a subset of the peer group as a golden peer group operating at a high standard of accuracy and precision;

a target module operating in conjunction with the status memory for determining from the golden peer group a set of periodic moving targets, the target module including means for automatically producing horizontal robustness across periods and vertical robustness across instruments in the golden peer group so as to automatically minimize, without human editing, the effect of outliers and erroneous data;

a correlation comparator comprising means for comparing the periodic control data from the data storage module with the periodic targets for a given interval comprising a number of periods to produce a CCC rating for said interval for each instrument in the peer group;

a CCC evaluator for determining the distribution of the CCC'"'"'s of all of the instruments in the peer group and rating the CCC of each instrument against said distribution; and

means responsive to the CCC evaluator for selecting the golden peer group from the peer group, and for updating the status memory with the golden peer group identification.

16. The system according to claim 15 wherein the CCC evaluator determines the distribution in terms of benchmark average and standard deviations for all instruments in the peer group, and the means responsive to the CCC evaluator selects all instruments within a predetermined standard deviation of the average as the golden peer group.

17. The system according to claim 16 wherein the predetermined standard deviation is about 1.

18. The system according to claim 15 wherein the control data produced on a periodic basis by each instrument comprises at least high and low data points spanning a desired operating range, and the set of periodic moving targets comprises at least high and low targets for each period.

19. The system according to claim 15 in which the periodic basis on which the control data is received is daily, and the interval over which the correlation comparator operates is on the order of monthly.

20. The system according to claim 19 in which the correlation comparator operates over dual intervals, one being a calendar month, and a second being the most recent d days comprising a "current" interval.

21. The system according to claim 20 in which the CCC evaluator updates the current interval daily, and the means responsive to the CCC evaluator updates the status memory over an interval substantially longer than daily.

22. The system according to claim 21 in which the means responsive to the CCC evaluator updates the status memory on about a monthly basis.

23. The system according to claim 21 in which the rating produced by the CCC evaluator for each instrument is reported to the instrument, such report being made within about a day or less of the receipt of the periodic control data.

24. The system according to claim 15 in which the target module further includes means for accumulating a set of fixed targets for the control data comprising an average of the periodic moving targets;

the correlation comparator also comprising means for comparing the fixed targets against the periodic control data for each instrument to determine for each laboratory a CCC for the interval with respect to the fixed targets; and

the CCC evaluator also comprising means for rating the fixed target CCC for each instrument against the distribution of CCC'"'"'s among all instruments in the peer group.

25. The system according to claim 15 in which the means for producing horizontal robustness across periods includes a moving median memory for storing n most recent periods of control data from each instrument, the moving median memory being operated to (a) store new control data for a new period, (b) delete the control data from the oldest period, and (c) determine and output the median for the new n periods and as the control data for that instrument for the new period.

26. The system according to claim 25 in which the period is daily, n is 7, and the moving median memory stores 7 days of control data sets for each instrument, the output for the most recent day being the median of the stored 7 days of control data.

27. The system according to claim 25 in which the means for producing vertical robustness across instruments includes a two-step M-estimation process for producing an average value from all instruments in the golden peer group for each period, the M-estimation process being performed on the periodic control data output from the moving median memory for each instrument in the golden peer group.

28. The system according to claim 15 in which a CCC memory stores the CCC of each instrument for each interval, the CCC evaluator comprising means for determining benchmark average and standard deviations of CCC'"'"'s among all instruments in the peer group for each interval, the means for rating comprising means for comparing the CCC for each instrument with the average and standard deviations determined from all instruments in the peer group.

29. The system according to claim 15 in which the means responsive to the evaluator comprises means for identifying all of the instruments within the peer group which have performed within a predetermined standard deviation of the average for at least the last i intervals, and means for marking said identified instruments as the golden peer group for subsequent evaluations.

30. The system according to claim 29 wherein the predetermined standard deviation is about 1.

31. The system according to claim 15 wherein the CCC for each instrument is determined against targets taken from only the golden peer group, but the CCC for each instrument is evaluated against the CCC'"'"'s taken from all instruments in the peer group, thereby to enhance the precision of the CCC while ranking the determined CCC against the entire peer group.

32. A method for quality control monitoring of a plurality of widely geographically dispersed instruments which are operated periodically to produce control data from common control samples, the method comprising the steps of:

electronically communicating the control data from each instrument to a central station on a periodic basis and automatically receiving and storing the control data at the central station, maintaining with the stored control data an identification of each instrument which produced it and the date on which it was produced;

producing a moving median periodic output for each instrument in a way which provides horizontal robustness by monitoring n periods and selecting for each new period median control data values for the new period which minimize the effect of outliers and erroneous data;

determining moving control targets for each period from the moving median periodic outputs by selecting a subset of the peer group comprising a golden peer group of good performing instruments and averaging the control data for each period across the golden peer group in a way which assures vertical robustness;

performing a concordance correlation comparison to produce a CCC for each instrument measuring its accuracy and precision with respect to the moving targets from the golden peer group over an interval comprising a number of periods;

determining the distribution of the CCC'"'"'s for all instruments in the peer group;

rating the CCC of each instrument for the interval against the distribution for all instruments in the peer group; and

selecting the golden peer group from the instruments near the center of the distribution.

33. The method as set forth in claim 32 wherein the step of rating the CCC of each instrument includes electronically communicating the rating to the instrument.

34. The method as set forth in claim 32 wherein the step of producing a moving median comprises storing in moving median memory control data from the last n periods, and reporting as the control data for the most recent period the median of the currently stored control data values.

35. The method as set forth in claim 34 wherein the step of producing a moving median further includes updating the moving median memory for a new period by (a) storing control data for the new period in the moving median memory, (b) deleting the oldest control data from the moving median memory, and (c) selecting the median of the stored control data values as the output for the new period.

36. The method as set forth in claim 32 further including the step of providing a fixed target which is the same for all periods within the interval, the fixed target comprising a composite of the moving targets of the golden peer group for the entire interval, and performing a CCC evaluation of the control data for each instrument for the interval against the fixed target to provide a CCC with respect to the fixed target.

37. The method as set forth in claim 32 wherein the golden peer group is smaller than and selected from the peer group, the CCC is determined from targets taken from only the golden peer group to increase the accuracy and precision of the CCC, but the CCC rating is evaluated against the CCC'"'"'s for all of the instruments in the peer group to provide a measure of the performance of each instrument with respect to the entire peer group.

38. The method as set forth in claim 32 wherein the period is daily, and each instrument reports at least high and low control points to the central station daily, the interval is monthly such that an evaluation for all instruments is produced monthly, and the memories accumulate the control data for the entire interval along with the control targets so that the CCC evaluation for all of the instruments can be performed on the last day of the interval after receipt of the periodic data from the last instrument in the peer group.

39. The method as set forth in claim 33 wherein the periodic basis on which the control data is communicated is daily and the interval over which the concordance correlation comparison is performed and the CCC'"'"'s rated is on the order of monthly.

40. The method as set forth in claim 39 in which the step of performing a concordance correlation comparison operates over two intervals, a first of said intervals comprising a calendar month, and a second of said intervals comprising the most recent d days for a "current" interval.

41. The method as set forth in claim 40 wherein the step of performing a concordance correlation comparison is performed daily to update the "current" interval daily, the step of rating being performed within about a day or less of receipt of the control data from the instruments.

42. The method as set forth in claim 41 wherein the step of selecting the golden peer group is performed about monthly.

43. A method of performing laboratory analyses on samples to produce results on a standardized scale, the scale being calibrated by periodically performing the analysis on known calibration samples, the method comprising the steps of obtaining a plurality of patient samples to be tested, performing the analysis on each of the plurality of patient samples to produce analytical reports for those samples, periodically performing the analysis on calibration samples to produce control data for the control samples and using the control data to validate the analytical reports, electronically reporting the control data to a central station along with like control data from all other laboratories making up a peer group which perform the same test and use the same control materials, at the central station:

storing all of the control data for a plurality of periods for the peer group;

selecting from the peer group a golden peer group of good performing laboratories;

setting control data targets against which to evaluate the laboratory control data, the control data targets being selected from only the golden peer group, and being selected in such a way as to assure horizontal and vertical robustness while automatically eliminating the effect of outliers and erroneous data;

for a given interval comprising a number of periods, comparing the control data from each laboratory against the targets determined from the golden peer group to produce a CCC for the interval for each laboratory;

combining the CCC'"'"'s from all of the laboratories in the peer group to determine the CCC distribution for the peer group;

rating each laboratory by the fit of its CCC into the distribution; and

utilizing the CCC evaluation to select the golden peer group.