Computer system and method for training certifying or monitoring human clinical raters

• US 20030125610A1
• Filed: 10/28/2002
• Published: 07/03/2003
• Est. Priority Date: 10/26/2001
• Status: Active Grant

• Current Assignee: Signant Health Global LLC
• Original Assignee: Concordant Rater Systems, LLC
• Inventors: DeBonis,, Daniel Vincent II, Perlis, Roy Howard, Sachs, Gary Steven
• Granted Patent: US 7,315,725 B2
• Time in Patent Office: Days
• US Class Current: 600/300
• CPC Class Codes: G09B 23/28 for medicine
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F
• CPC Class Codes: G16H 10/20 for electronic clinical trials or questionnaires
• CPC Class Codes: G16H 10/60 for patient-specific data, e.g. for electronic patient records
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: G16H 70/60 relating to pathologies

Abstract

A method and system are disclosed for computerized training, monitoring, certification or re-certification of human raters in clinical trials. The invention provides an interactive computerized interview that can be compared scores obtained by human raters tp monitor and train clinical raters on a continuing basis in order to reduce rater drift and variance during clinical trials. Remediation can be automatically flagged or provided to improve a deviating rater'"'"'s performance. Computerized assessment of symptom severity is provided without the need for human clinical raters. The system and method enable screening of prospective patients for inclusion or exclusion from a clinical trial by automatically obtaining computerized ratings of symptom severity.

Claims

1. A method for quantifying severity of a previously diagnosed condition in a subject through a computerized interview that comprises:

presenting a first question to the subject and receiving a first input from the subject in response thereto;

based on the first input, selecting a second question from a plurality of alternative second questions;

presenting the selected second question to the subject and receiving a second input from the subject in response thereto; and

based on the inputs received from the subject, determining in accordance with a rating scale a computerized score of the severity of the previously diagnosed condition in the subject.

2. The method according to claim 1, wherein the previously diagnosed condition is a psychiatric illness.

3. The method according to claim 1, wherein the rating scale is in accordance with one of the following scales:

the Young Mania Rating Scale (YMRS);

the Montgomery-Asberg Depression Rating Scale (MADRS);

the Hamilton Depression Scale (HAMD);

the Schedule for Affective Disorders and Schizophrenia, Current symptom version mania Rating Scale (SADS-C);

an Additional Psychiatry Scale;

a Geriatric/Cognitive Scale;

an Other Cognitive Assessment;

a Non-CNS Scale;

or a member of the APA group of scales.

4. The method according to claim 1, wherein at least one of the unselected alternative second questions is not presented to the subject.

5. The method according to claim 1, further comprising:

based on the second input, selecting a third question from a plurality of alternative third questions; and

presenting the selected third question to the subject and receiving a third input from the subject in response thereto.

6. The method according to claim 5, wherein at least one of the unselected alternative second questions and at least one of the unselected alternative third questions are not presented to the subject.

7. The method according to claim 1, wherein the questions are presented by visual, audio, or multimedia display.

8. The method according to claim 1, wherein the inputs are received directly from the subject by means of a keyboard, a pointing device, or a touch-screen device or voice response.

9. The method of claim 1, further comprising:

automatically determining by previously established criteria whether or not the subject qualifies for inclusion in a clinical trial.

10. A method for recertifying or monitoring performance of a clinical rater comprising:

receiving one or more rater inputs reflecting the rater'"'"'s evaluation of severity of a previously diagnosed condition in a subject at a clinical interview;

performing a computerized assessment of the subject in accordance with the method of claim 1 to obtain a computerized score; and

determining whether the one or more rater inputs are in accordance with the computerized score.

11. The method according to claim 10, wherein the computerized assessment is performed within 24 hours after the clinical interview.

12. The method according to claim 11, wherein the clinical interview and computerized assessment are performed during the same clinic visit.

13. A method for recertifying or monitoring performance of a clinical rater comprising:

receiving a plurality of rater inputs reflecting a given rater'"'"'s evaluation of severity of a previously diagnosed condition in a plurality of subjects at a plurality of clinical interviews;

performing computerized assessments of the subjects in accordance with the method of claim 1 to obtain a plurality of computerized scores; and

determining whether the rater inputs are in accordance with the computerized scores.

14. A method for recertifying or monitoring performance of a clinical rater comprising:

receiving a plurality of rater inputs reflecting a given rater'"'"'s evaluation of severity of a previously diagnosed illness in a subject at a plurality of clinical interviews of that subject;

in conjunction with the clinical interviews, obtaining computerized assessments of the subject by performing the method according to claim 1 to obtain a plurality of computerized scores; and

determining whether the rater inputs are in accordance with the computerized scores.

15. A method for improving performance of a clinical rater, comprising:

monitoring performance of the clinical rater in accordance with the method of claim 10;

and

if the first rater inputs are not in accordance with the computerized score, providing an indication that the rater is in need of remedial training.

16. A method for improving performance of a clinical rater, comprising:

monitoring performance of the clinical rater in accordance with the method of claim 10;

and

if the first rater inputs are not in accordance with the computerized score, providing remedial training to the rater.

17. The method of claim 16, wherein the remedial training includes explanatory material concerning how to evaluate severity of the previously diagnosed condition.

18. The system of claim 16 wherein the remedial training includes presenting a remedial script or a remedial semi-script to the rater.

19. A computer programmed to perform the method according to claim 1.

20. A computer readable medium having stored thereon instructions for causing a computer to perform the method according to claim 1.

21. A system for performing the method according to claim 1, comprising:

an output device configured to present selected questions to the subject;

an input device configured to accept inputs from the subject; and

one or more computers configured to present selected questions to the subject by means of the output device, receive inputs from the subject by means of the input device, and determine the computerized score based on the inputs received from the subject in accordance with the rating scale.

22. A computerized method for training, certifying or recertifying a plurality of clinical raters, comprising:

presenting to a plurality of clinical raters one or more stored segments of an actual or simulated patient interview or interviews;

eliciting from the raters one or more inputs in response to presentation of the one or more stored segments; and

for individual raters, comparing the one or more inputs of that rater by means of a computer with one or more reference ratings to determine whether that rater'"'"'s input accords with the reference rating.

23. The method according to claim 22, wherein the steps of presenting and eliciting comprise one or more of the following cycles for individual raters:

presenting to the rater a first stored segment of an actual or simulated patient interview;

eliciting from that rater a first input in response to presenting the first stored segment; and

in response to receiving the first input from the rater, presenting to that same rater a second stored segment of an actual or simulated patient interview,

wherein presentation of the second stored segment is triggered for individual raters in response to receiving the input from that rater in response to the first stored segment.

24. The method according to claim 22, wherein individual raters are provided with non-synchronous sessions for training, certification or recertification.

25. The method according to claim 22, further comprising:

based on the comparison, displaying an indication of the reference rating and associated rationale.

26. The method of claim 22, wherein the one or more stored segments include a video segment, an audio segment, or a multimedia segment.

27. The method of claim 22, wherein the one or more reference ratings are determined by an expert or reflect a consensus of a plurality of experts.

28. The method of claim 22 wherein the one or more stored reference ratings are determined by intra-class correlation of inputs elicited from a plurality of raters.

29. The method of claim 22, wherein the one or more reference ratings are in accordance with a scale for rating severity of symptoms of a medical condition.

30. The method of claim 29, wherein the medical condition is a psychiatric illness.

31. The method of claim 29, wherein the symptom severity rating scale is in accordance with one of the following scales:

the Young Mania Rating Scale (YMRS);

the Montgomery-Asberg Depression Rating Scale (MADRS);

the Hamilton Depression Scale (HAMD);

the Schedule for Affective Disorders and Schizophrenia, Current symptom version mania Rating Scale (SADS-C);

an Additional Psychiatry Scale;

a Geriatric/Cognitive Scale;

an Other Cognitive Assessment;

a Non-CNS Scale;

or a member of the APA group of scales.

32. A computer programmed to perform the method according to claim 22.

33. A computer readable medium having stored thereon instructions for causing a computer to perform the method according to claim 22.

34. A system for performing the method according to claim 22, comprising:

one or more storage media having stored therein the one or more segments of an actual or simulated patient interview or interviews;

one or more output devices configured to present the one or more segments to the clinical raters;

one or more input devices configured to accept the one or more inputs from the raters; and

one or more computers configured to compare the one or more rater inputs with the one or more reference ratings and to determine whether a rater'"'"'s input accords with the reference rating.

35. A computerized method for assisting or training a rater to evaluate severity of symptoms in a subject affected by a medical condition, comprising:

presenting a first output to the rater conveying a first suggestion concerning a question to be posed to the subject or a clinical observation to be made concerning the subject;

receiving a first input from the rater in response to the first display;

based on the first input, selecting a second output from a plurality of alternative second outputs concerning a question to be posed to the subject or a clinical observation to be made concerning the subject and presenting the second output to the rater

36. The method according to claim 35, further comprising:

based on the inputs received from the rater, determining in accordance with a rating scale a computerized score of the severity of the medical condition.

37. The method according to claim 35, wherein the rater is a health care provider evaluating the severity of the symptoms in a patient or evaluating a response of a patient to treatment or placebo.

38. A computer programmed to perform the method according to claim 35.

39. A computer readable medium having stored thereon instructions for causing a computer to perform the method according to claim 35.

40. A system for performing the method according to claim 35, comprising:

an output device configured to present selected outputs to the rater;

an input device configured to accept inputs from the rater; and

one or more computers configured to present selected outputs to the health care provider by means of the output device and receive inputs from the rater by means of the input device.