Computer architecture and process of user evaluation

• US 8,096,811 B2
• Filed: 11/29/2004
• Issued: 01/17/2012
• Est. Priority Date: 11/29/2003
• Status: Active Grant

• Current Assignee: American Board of Family Medicine, Inc.
• Original Assignee: American Board of Family Medicine, Inc.
• Inventors: Sumner, Walton II, Xu, Jinzhong, Roussel, Guy H., Rovinelli, Richard J., Hagen, Michael D.
• Primary Examiner(s): Gishnock, Nikolai A
• Application Number:
  US10/998,384
• Publication Number:
  US 20050118557A1
• Time in Patent Office: 2,605 Days
• Field of Search: 434/262, 434/362, 706/15, 706/924
• US Class Current: 434/262
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

The present invention is generally related to a computer architecture and process for evaluating and reporting on a user'"'"'s competence in managing a simulation of a complex system modeled in a computer based testing and training system. The simulation may be called a “virtual system.” The method generates evaluation criteria responsive to a user profile, user choices in managing the virtual system, and events that occur during a simulation, in addition to the original state of the system. The method produces summary evaluations responsive to a plurality of characteristics of the simulated system. The implementation may be advantageously employed to evaluate a medical clinician'"'"'s management of computer-simulated human patients, hereinafter called “virtual patients”. The present invention is also suitable for evaluation of user management of a plurality of complex systems including but not limited to animals, plants, cells, machines, and populations.

Claims

1. A computer implemented process for evaluating user interactions with a dynamic simulation of a system portraying at least one system state and responsive to user queries and interventions comprising the steps of:

a) assigning a simulation to at least one user;

b) using a computer to generate a simulated system comprising a system age and at least one system state affecting said system;

c) using a computer to dynamically generate at least one criterion, said criterion associated with at least one of said system state, a query, a query result, and an intervention, when a change occurs in said at least one system state affecting said system by;

i) identifying an inference program;

ii) providing at least one input datum relating to said change in said at least one system state affecting said system, said at least one input datum defining the state of an input node in said inference program;

iii) generating at least one criterion as an inference from said at least one input datum;

said criterion comprising;

a) a selected user activity to monitor, said activity comprising starting an intervention to improve the present or future state of the simulated system;

b) a system age range in which to monitor said user activity, said age range comprising a start age and a stop age, wherein said start age is calculated by adding said system age to a relative start time, and said stop age is calculated by adding said system age to a relative stop time;

c) a classification of said user activity as desirable or not desirable; and

d) a weight value selected from a weight value set having at least one weight value;

d) accepting at least one user action comprised of said selected user activity and an age of the simulated system when the selected user activity occurs;

e) evaluating said at least one accepted user action to determine if at least one of said accepted user action and its consequences require generation of additional criterion and repeating steps c and d if additional criterion require generation;

f) determining a status of said at least one criterion dynamically generated in step c; and

,

g) generating at least one performance description based on at least one of;

desirable actions performed;

undesirable actions avoided;

desirable actions omitted, and undesirable actions committed.

2. The computer implemented process for evaluating user interactions of claim 1, wherein said at least one criterion dynamically generated in step c identifies an abstract monitored user activity representing a plurality of user activities that define actual user actions that the simulation accepts in step d.

3. The computer implemented process for evaluating user interactions of claim 1, where, after the step of generating at least one performance description, the process comprises the further step of:

reporting said at least one performance description.

4. The computer implemented process for evaluating user interactions of claim 3, wherein the step of reporting said at least one performance description comprises the substeps of:

a) obtaining a feedback phrase associated with the status of at least one criterion;

b) generating a report summarizing at least one of;

desirable actions performed;

undesirable actions avoided;

desirable actions omitted, and undesirable actions committed; and

c) presenting said report.

5. The computer implemented process for evaluating user interactions of claim 4, where said at least one criterion dynamically generated includes a dynamically generated explanation, said dynamically generated explanation being a user-interpretable description of said at least one criterion and said dynamically generated explanation being included as a preamble to said generated report.

6. The computer implemented process for evaluating user interactions of claim 1, where said system age range is dynamically dependent on an association between said simulated system and with at least one of one said system state, a query, a query result, and an intervention.

7. The computer implemented process for evaluating user interactions of claim 1, where a dynamically generated criterion may further comprise a specification for presenting at least one direct question to said assigned user, said specification comprising a dynamically generated stem and at least one response.

8. The computer implemented process for evaluating user interactions of claim 1, where a dynamically generated criterion may further comprise a specification for presenting at least one direct question to said assigned user, said specification comprising:

a) a stem comprised of a dynamically generated explanatory component and a dynamically generated media component;

b) a dynamically generated boundary specification enclosing a portion of the dynamically generated media component;

c) a first quantification identifying what portion of the boundary specification that the assigned user must include in a subsequently identified user boundary; and

,

d) a second quantification identifying what portion of said subsequently identified user boundary that the boundary specification must include.

9. The computer implemented process for evaluating user interactions of claim 8, where said step of determining a status classifies said dynamically generated criterion as met only where said first quantification is exceeded and where said second quantification is not exceeded and otherwise classifies said dynamically generated criterion as not met.

10. The computer implemented process for evaluating user interactions of claim 9, where said specification for presenting at least one direct question to said assigned user further comprises a first instruction to follow where said status is determined as met and a second instruction to follow where said status is determined as not met.

11. The computer implemented process for evaluating user interactions of claim 1, where a dynamically generated criterion may further comprise a specification for presenting at least one direct question to said assigned user, where said direct question requires said assigned user to select a media portrayal, said specification comprising:

a) a dynamically generated stem; and

,

b) a plurality of media responses, each said media response comprising;

dynamically generated media responsive to the system context and a numeric score.

12. The computer implemented process for evaluating user interactions of claim 1, where a dynamically generated criterion may further comprise a specification for presenting at least one direct question to said assigned user, where said direct question requires said assigned user to critique at least one media portrayal, said specification comprising:

a) a dynamically generated stem;

b) at least one media item, said media item comprising;

dynamically generated media responsive to the system context and at least one observation including at least one identifying label and a value; and

c) a scoring formula for combining values of observations to generate a score.

13. The computer implemented process of claim 1 further comprising dynamically generating at least one criterion when one of a system state, query, and intervention begins by:

a) identifying an inference program;

b) providing at least one input datum relating to said one of a system state, query, and intervention, said at least one input datum defining the state of an input node in said inference program;

c) solving the inference program to generate at least one criterion as an inference from the at least one input datum.

14. The computer implemented process of claim 1, wherein determining a status of said at least one criterion in step f further comprises identifying said at least one criterion as fulfilled if:

a) said at least one accepted user action is identical to or a descendent of said selected user activity; and

b) said age of said simulated system when said selected user activity occurs is between said start age and said stop age.