Computer architecture and process of user evaluation

• US 20050118557A1
• Filed: 11/29/2004
• Published: 06/02/2005
• Est. Priority Date: 11/29/2003
• Status: Active Grant

• Current Assignee: American Board of Family Medicine, Inc.
• Original Assignee: American Board of Family Medicine, Inc.
• Inventors: Xu, Jinzhong, Hagen, Michael D., Roussel, Guy H., Sumner, Walton II, Rovinelli, Richard J.
• Granted Patent: US 8,096,811 B2
• Time in Patent Office: Days
• US Class Current: 434/118
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

The present invention is generally related to a computer architecture and process for evaluating and reporting on a user'"'"'s competence in managing a simulation of a complex system modeled in a computer based testing and training system. The simulation may be called a “virtual system.” The method generates evaluation criteria responsive to a user profile, user choices in managing the virtual system, and events that occur during a simulation, in addition to the original state of the system. The method produces summary evaluations responsive to a plurality of characteristics of the simulated system. The implementation may be advantageously employed to evaluate a medical clinician'"'"'s management of computer-simulated human patients, hereinafter called “virtual patients”. The present invention is also suitable for evaluation of user management of a plurality of complex systems including but not limited to animals, plants, cells, machines, and populations.

Claims

1. A computer implemented process for evaluating user interactions with a dynamic simulation of a system portraying at least one system state and responsive to user queries and interventions comprising the steps of:

a. assigning a simulation to at least one user;

b. generating a simulated system comprising a system age and at least one system state affecting said system;

c. dynamically generating at least one criterion associated with at least one of one said system state, a query, a query result, and an intervention;

said criterion comprising;

i. a selected user activity to monitor, said activity selected from the group comprising;

a. submitting a query to obtain information from the simulated system,

b. identifying a state portrayed by the simulated system,

c. starting an intervention to improve the present or future state of the simulated system, or

d. arranging a disposition comprising scheduling a subsequent interaction between the simulated system and one of said at least one users and an expert in managing a system state;

ii. a system age range in which to monitor said user activity, said age range comprising a start age and a stop age;

iii. a classification of said user activity as desirable or not desirable; and

iv. a weight value selected from a weight value set having at least one weight value;

d. accepting at least one user action comprised of said selected user activity and an age of the simulated system when the selected user activity occurs;

e. evaluating said at least one accepted user action to determine if at least one of said accepted user action and its consequences require generation of additional criterion and repeating steps c and d if additional criterion require generation;

f. determining a status of said at least one criterion dynamically generated in step c; and

,

g. generating at least one performance description based on at least one of;

desirable actions performed;

undesirable actions avoided;

desirable actions omitted, and undesirable actions committed.

2. The computer implemented process for evaluating user interactions of claim 1, wherein the step of dynamically generating at least one criterion comprises the substeps of:

a. generating at least one inference program requiring as inputs a plurality of possible characteristics of said simulated system;

b. generating a plurality of actual characteristics of said simulated system as inputs to the inference program; and

c. generating at least one criterion inferred by the inference program from the input characteristics of the simulated system.

3. The computer implemented process for evaluating user interactions of claim 1, wherein the step of dynamically generating at least one criterion comprises the substeps of:

a. generating at least one Bayesian network describing a plurality of possible characteristics of said simulated system;

b. reconciling said at least one Bayesian network with a plurality of actual characteristics of said simulated system; and

c. generating at least one criterion associated with the state of at least one Bayesian node of said at least one Bayesian network.

4. The computer implemented process for evaluating user interactions of claim 1, wherein said at least one criterion dynamically generated in step c identifies an abstract monitored user activity representing a plurality of user activities that define actual user actions that the simulation accepts in step d.

5. The computer implemented process for evaluating user interactions of claim 1, where, after the step of generating at least one performance description, the process comprises the further step of:

reporting said at least one performance description.

6. The computer implemented process for evaluating user interactions of claim 5, wherein the step of reporting said at least one performance description comprises the substeps of:

a. obtaining a feedback phrase associated with the status of at least one criterion.

b. generating a report summarizing at least one of;

desirable actions performed;

undesirable actions avoided;

desirable actions omitted, and undesirable actions committed; and

c. presenting said report.

7. The computer implemented process for evaluating user interactions of claim 6, where said at least one criterion dynamically generated includes a dynamically generated explanation, said dynamically generated explanation being included as a preamble to said generated report.

8. The computer implemented process for evaluating user interactions of claim 1, where said system age range is dynamically dependent on an association between said simulated system and with at least one of one said system state, a query, a query result, and an intervention.

9. The computer implemented process for evaluating user interactions of claim 1, where a dynamically generated criterion may further comprise a specification for presenting at least one direct question to said assigned user, said specification comprising a dynamically generated stem and at least one response.

10. The computer implemented process for evaluating user interactions of claim 1, where a dynamically generated criterion may further comprise a specification for presenting at least one direct question to said assigned user, said specification comprising:

a. a stem comprised of a dynamically generated explanatory component and a dynamically generated media component;

b. a dynamically generated boundary specification enclosing a portion of the dynamically generated media component;

c. a first quantification identifying what portion of the boundary specification that the assigned user must include in a subsequently identified user boundary; and

,

d. a second quantification identifying what portion of said subsequently identified user boundary that the boundary specification must include.

11. The computer implemented process for evaluating user interactions of claim 10, where said step of determining a status classifies said dynamically generated criterion as met only where said first quantification is exceeded and where said second quantification is not exceeded and otherwise classifies said dynamically generated criterion as not met.

12. The computer implemented process for evaluating user interactions of claim 11, where said specification for presenting at least one direct question to said assigned user further comprises a first instruction to follow where said status is determined as met and a second instruction to follow where said status is determined as not met.

13. The computer implemented process for evaluating user interactions of claim 1, where a dynamically generated criterion may further comprise a specification for presenting at least one direct question to said assigned user, where said direct question requires said assigned user to select a media portrayal, said specification comprising:

a. a dynamically generated stem; and

,

b. a plurality of media responses, each said media response comprising;

dynamically generated media responsive to the system context and a numeric score.

14. The computer implemented process for evaluating user interactions of claim 1, where a dynamically generated criterion may further comprise a specification for presenting at least one direct question to said assigned user, where said direct question requires said assigned user to critique at least one media portrayal, said specification comprising:

a. a dynamically generated stem;

b. at least one media item, said media item comprising;

dynamically generated media responsive to the system context and at least one observation including at least one identifying label and a value; and

c. a scoring formula for combining values of observations to generate a score.

15. A computer implemented process for evaluating user interactions with a dynamic simulation of a system portraying at least one system state and responsive to user queries and interventions comprising the steps of:

a. assigning a simulation to at least one user;

b. generating a simulated system comprising a system age and at least one state affecting said system;

c. accepting at least one user action by said user, said user action selected from the group comprising;

i. submitting a query to obtain information from the simulated system;

ii. proposing an explanation of a perceived state of the simulated system;

iii. starting an intervention to improve a present or future state of the simulated system;

or

iv. arranging a disposition comprising scheduling a subsequent interaction between the simulated system and one of the assigned user and an expert in managing a system state;

d. associating at least one of a system state and said at least one accepted user action with at least one inference program;

e. evaluating at least one criterion by said at least one inference program, said evaluating comprising the steps of;

i. comparing said at least one criterion to said accepted at least one user action;

ii. determining a status of said at least one criterion;

iii. entering said status of said at least one criterion as input data to said inference program;

iv. entering patient data as input to said inference program; and

v. obtaining at least one performance description as an output from said inference program.

16. The computer implemented process for evaluating user interactions of claim 15, where the at least one inference program of step d is a Bayesian Network.

17. The computer implemented process for evaluating user interactions of claim 15, wherein said at least one criterion dynamically generated in step e identifies an abstract user activity representing a plurality of user activities that define actual user actions that the simulation accepts in step c.

18. The computer implemented process for evaluating user interactions of claim 15, where, after the step of generating at least one performance description, the process comprises the further step of:

reporting said at least one performance description.

19. The computer implemented process for evaluating user interactions of claim 15, wherein one of said at least one performance description is a report to the user.

20. The computer implemented process for evaluating user interactions of claim 19, wherein a dynamically generated explanation is included as a preamble to said generated report.

21. The computer implemented process for evaluating user interactions of claim 15, where step e.ii., determining a status of said at least one criterion, includes determining a system age range, said system age range being dynamically dependent on an association between said simulated system and at least one of one said system state, a query, a query result, and an intervention.

22. The computer implemented process for evaluating user interactions of claim 15, wherein said at least one criterion may further comprise a specification for presenting at least one direct question to said assigned user, said specification comprising a dynamically generated stem and at least one response.

23. The computer implemented process for evaluating user interactions of claim 15, wherein said at least one criterion may further comprise a specification for presenting at least one direct question to said assigned user, said specification comprising:

a. a stem comprised of a dynamically generated explanatory component and a dynamically generated media component;

b. a dynamically generated boundary specification enclosing a portion of the dynamically generated media component;

c. a first quantification identifying what portion of the boundary specification that the assigned user must include in a subsequently identified user boundary; and

,

d. a second quantification identifying what portion of said subsequently identified user boundary that the boundary specification must include.

24. The computer implemented process for evaluating user interactions of claim 23, where said at least one criterion as met only where said first quantification is exceeded and where said second quantification is not exceeded and otherwise classifies said dynamically generated criterion as not met.

25. The computer implemented process for evaluating user interactions of claim 24, where said specification for presenting at least one direct question to said assigned user further comprises a first instruction to follow where said status is determined as met and a second instruction to follow where said status is determined as not met.

26. The computer implemented process for evaluating user interactions of claim 15, where said at least one criterion may further comprise a specification for presenting at least one direct question to said assigned user, where said direct question requires said assigned user to select a media portrayal, said specification comprising:

a. a dynamically generated stem; and

,

b. a plurality of media responses, each said media response comprising;

dynamically generated media responsive to the system context and a numeric score.

27. The computer implemented process for evaluating user interactions of claim 15, where said at least one criterion may further comprise a specification for presenting at least one direct question to said assigned user, where said direct question requires said assigned user to critique at least one media portrayal, said specification comprising:

a. a dynamically generated stem;

b. at least one media item, said media item comprising;

dynamically generated media responsive to the system context and at least one observation including at least one identifying label and a value; and

c. a scoring formula for combining values of observations to generate a score.

28. A computer implemented process for evaluating user interactions with a multi-factoral system simulation wherein at least one recursively defined evaluation structure coordinates an assessment of said user interactions, comprising the steps of:

a. identifying at least one evaluation structure linked to at least one of a complex system, a state of said complex system, a query that reveals information regarding said complex system, a specific response generated by said query, and an intervention;

b. obtaining recursively from said evaluation structure at least one of an inference program and a list of subordinate evaluation structures;

c. collecting a plurality of said inference programs by recursive inspection of evaluation structures;

d. using said plurality of inference programs identified in step c to dynamically evaluate the user interaction with the system simulation;

e. calculating for each of said plurality of inference programs at least one performance description based on the user interaction with the simulation; and

f. associating said at least one performance description with at least one assessment theme selected from a set of at least one assessment themes.

29. The computer implemented process of claim 28 where a plurality of performance descriptions related to said at least one assessment theme are combined to obtain a summary result for user performance related to said at least one assessment theme.

30. The computer implemented process of claim 29 wherein an inference program combines said plurality of performance descriptions.

31. The computer implemented process of claim 30 wherein a Bayesian network combines the performance descriptions.

32. The computer implemented process of claim 29 wherein the combination process anticipates arbitrary numbers of performance descriptions.

33. The computer implemented process of claim 28 where at least one of said at least one performance description is associated with an assessment theme.

34. The computer implemented process of claim 28 where at least one of said at least one performance description is associated with a retesting interval.

35. The computer implemented process of claim 28 where at least one of said at least one performance description is associated with a retesting interval and an assessment theme.