Real-time feedback of task performance

• US 9,754,512 B2
• Filed: 09/27/2010
• Issued: 09/05/2017
• Est. Priority Date: 09/30/2009
• Status: Active Grant

• Current Assignee: Augusta University Research Institute, Inc. (University System of Georgia)
• Original Assignee: University of Florida Research Foundation, Incorporated (State University System of Florida), Augusta University Research Institute, Inc. (University System of Georgia)
• Inventors: Kotranza, Aaron Andrew, Lok, Benjamin Chak Lum, Lind, David Scott
• Primary Examiner(s): Utama, Robert J
• Assistant Examiner(s): Bullington, Robert P
• Application Number:
  US13/499,161
• Publication Number:
  US 20120225413A1
• Time in Patent Office: 2,535 Days
• Field of Search: 434247, 434262
• CPC Class Codes: G09B 23/285 for injections, endoscopy, bronchoscopy, sigmoidscopy, insertion of contraceptive devices or enemas
• CPC Class Codes: G09B 23/30 Anatomical models G09B23/281 - G09B23/288 take precedence

Abstract

A method of providing real-time feedback of task performance includes receiving sensor data from sensors coupled to an object. The sensor data is received in response to tasks performed on the object and references positional data with respect to a location on the object at which tasks are performed. The method also includes applying the sensor data to a model of performance measurements, which reflect a standard of performance. The method further includes identifying any deviations of the sensor data from data in the model, and outputting feedback in real-time. The feedback includes indicators signaling tasks performed are correctly implemented and indicators signaling tasks performed are incorrectly implemented. Corrective indicators include measures for modifying task performance to achieve an outcome that approaches or meets the standard of performance. The indicators correspond to one or more locations on the object as determined by the positional data.

Claims

1. A method of providing real-time instructional information relating to task performance, comprising:

receiving sensor data from sensors coupled to a physical object, the sensor data received at a computer in response to a task performed on the physical object by a subject, the sensor data referencing positional data with respect to a location on the physical object to which the task is performed;

applying the sensor data to a model of performance with respect to the task;

receiving, at the computer, video data of the subject performing the task;

calculating at least one performance value resulting from applying the sensor data to the model; and

outputting instructional information in real-time to the subject via at least one output device, wherein the computer derives the instructional information based on the at least one performance value and the video data such that the instructional information comprises;

indicators signaling whether the task was performed on the physical object in accordance with the model; and

a representation of a human exhibiting a behavior that is a response to performance of the task by the subject on the human.

2. The method of claim 1, wherein the physical object is a simulated anatomy of a patient that is being examined, and the task represents examination of the simulated anatomy by the subject.

3. The method of claim 2, wherein the examination is a clinical breast examination and the sensors are configured to measure physical qualities of the examination for which the instructional information is to be provided.

4. The method of claim 2, wherein outputting the instructional information in real-time further includes displaying a computer-generated simulated human exhibiting a human response to produce a mixed environment, wherein the mixed environment includes at least the physical object and the video data.

5. The method of claim 4, further comprising:

receiving sensor data for the task performed on the physical object by another subject, the sensor data reflecting qualities of the task performed by the other subject; and

calculating numerical models of examination performance by the other subject to produce a standard of performance.

6. The method of claim 1, wherein outputting the instructional information in real-time includes:

displaying, for the subject via the output device, a video representation of the task performed by the subject with respect to the physical object; and

overlaying, on the video representation, real-time in-situ feedback of the task performed by the subject.

7. The method of claim 6, wherein the positional data further comprises location data with respect to a continuous sequence of tasks forming a pattern, and wherein applying the sensor data to the model includes:

applying the pattern of the positional data to a pattern of positional data in the model, and calculating quantitative performance values that include any deviations resulting from the applying the pattern of the positional data to the model, the deviations identified as an angle between matched segments of the patterns of positional data for the respective subject and model;

wherein overlaying the real-time in-situ instructional information includes simultaneously displaying a pattern of search map created by the model overlaid by a pattern of search map representing the continuous sequence of tasks forming the pattern.

8. The method of claim 7, wherein the indicators signaling whether the task was performed on the physical object in accordance with the model comprise displaying a color pattern on the physical object, with different colors depicting conformance and non-conformance to the model.

9. The method of claim 1, wherein the sensors are force sensors and the receiving sensor data further includes receiving voltage values representing pressure applied on the physical object by the subject, the method further comprising:

applying the voltage values to the model; and

identifying any deviations of the sensor data specifying the voltage values from data in the model.

10. The method of claim 9, wherein outputting the instructional information in real-time includes:

displaying, for the subject via the output device, a video representation of the task performed by the subject with respect to the physical object; and

overlaying, on the video representation, real-time in-situ instructional information of the task performed by the subject;

wherein the real-time in-situ instructional information of the task includes displaying a sequence of visually-distinct elements representing the pressure data, the positional data, and a coverage area specifying physical locations of the object in which the task has been performed.

11. A system for providing real-time instructional information relating to task performance, the system comprising:

a computer;

sensors coupled to a physical object and in communication with the computer, wherein the physical object comprises, at least in part a simulated human anatomy; and

an output device in communication with the computer, wherein the computer is configured to implement a method comprising;

receiving sensor data from the sensors, the sensor data received in response to a task performed on the physical object by a subject, the sensor data referencing positional data with respect to a location on the physical object to which the task is performed;

determining based on the sensor data an indication of performance of the task by comparing the sensor data to a model of performance;

determining, based on the indication of performance, at least one of an audio feedback element and a visual feedback element that is a representation of a human exhibiting a behavior responding to the task being performed on the human; and

outputting, to the subject via the output device in real-time during performance of the task by the subject on the physical object, the representation of the human.

12. The system of claim 11, wherein the task represents a medical examination of the simulated anatomy by the subject.

13. The system of claim 12, wherein the examination is a clinical breast examination and the sensors are configured to measure physical qualities of the examination, and the computer is further configured to provide feedback related to appropriateness of pressure provided during the clinical breast examination.

14. The system of claim 11, further comprising a camera, wherein outputting the representation of the human in real-time further includes outputting a video stream with a computer-generated simulated extension of the simulated human anatomy to produce a mixed environment, the simulated extension being based at least in part on information captured with the camera.

15. The system of claim 14, wherein:

the system further comprises an object associated with the simulated human anatomy;

the computer determines the human behavior based at least in part on a position of the object determined from the information captured with the camera.

16. The system of claim 11, wherein outputting the representation of the human in real-time includes:

displaying, for the subject via the output device, computer-generated, interactive virtual human exhibiting behaviors selected based on the indication of performance.

17. The system of claim 16, wherein the positional data further comprises location data with respect to a continuous sequence of actions forming a pattern, and wherein determining based on the sensor data comprises applying the sensor data to a model by:

applying the pattern of the positional data to a pattern of positional data in the model, and calculating quantitative performance values that include any deviations resulting from the applying the pattern of the positional data to the model, the deviations identified as an angle between matched segments of the patterns of positional data for the respective subject and model.

18. The system of claim 17, wherein the physical object comprises indicators, and the computer is further configured to signal whether the task was performed on the physical object in accordance with the model comprise displaying with the indicators a color pattern on the physical object, with different colors depicting conformance and non-conformance to the model.

19. The system of claim 11, wherein the sensors are force sensors and the receiving sensor data further includes receiving voltage values representing pressure applied on the physical object by the subject, the method further comprising:

applying the voltage values to the model; and

identifying any deviations of the sensor data specifying the voltage values from data in the model.

20. The system of claim 19, wherein the method further comprises outputting the instructional information in real-time by:

displaying, for the subject via the output device, a video representation of the task performed by the subject with respect to the physical object; and

overlaying, on the video representation, real-time in-situ feedback of the task performed by the subject;

wherein the instructional information comprises real-time in-situ feedback comprising a sequence of visually-distinct elements representing the pressure data, the positional data, and a coverage area specifying physical locations of the object in which the task has been performed.

21. A method of providing a mixed environment for learning a medical procedure, wherein the mixed environment comprises at least one physical object representing a patient with which a subject interacts physically and at least one computer generated display visible to the subject, the method comprising:

determining physical interactions between the subject and the at least one physical object during performance of a simulated medical procedure;

with at least one computing device, assessing performance of the medical procedure by the subject based on comparing the determined physical interactions to a model of performance, and determining, based on the assessed performance of the medical procedure by the subject, at least one feedback element representing behavior of a human in response to the physical interactions; and

concurrently rendering to the subject;

instructional information based on the assessed performance; and

the at least one feedback element as simulated behavior of the patient during performance of the medical procedure.

22. The method of claim 21, wherein:

the method further comprises;

receiving, via a camera, video of performance of the medical procedure by the subject;

determining the physical interactions comprises measuring pressure on the physical object with pressure sensors; and

applying data from the pressure sensors and information extracted from the video to a computer model of performance of the medical procedure to determine the assessed performance; and

concurrently rendering to the subject an indication of the assessed performance and simulated behavior of a human patient during performance of the task comprises presenting a mixed learning environment with a virtual human agent to the subject.

23. The method of claim 21, further comprising:

tracking, via a camera, a marker attached to a movable object within the mixed environment.

24. The method of claim 23, wherein the movable object comprises an article of patient clothing.

25. The method of claim 21, wherein:

determining the physical interactions comprises;

measuring pressure on the physical object with pressure sensors; and

computing interpolated sensor data based on the received sensor data and at least one sensor distribution function.

26. The method of claim 21, wherein

the method further comprises generating the model of performance based on sensor data recorded during performance of the task by an expert.