Video-Based System for Improving Surgical Training by Providing Corrective Feedback on a Trainee's Movement

• US 20140220527A1
• Filed: 02/06/2014
• Published: 08/07/2014
• Est. Priority Date: 02/07/2013
• Status: Abandoned Application

• Current Assignee: Arizona Board of Regents (University of Arizona)
• Original Assignee: Arizona Board of Regents (University of Arizona)
• Inventors: Li, Baoxin, Zhang, Qiang, Zhang, Peng, Chen, Lin
• Application Number:
  US14/174,372
• Publication Number:
  US 20140220527A1
• Time in Patent Office: Days
• US Class Current: 434/262
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/84 using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G09B 19/003 Repetitive work cycles; Sequence of movements
• CPC Class Codes: G09B 23/285 for injections, endoscopy, bronchoscopy, sigmoidscopy, insertion of contraceptive devices or enemas
• CPC Class Codes: G09B 9/00 Simulators for teaching or training purposes for the use of weapons F41; computing aspects G06; protocols for games, networked simulations or virtual reality H04L67/131

Abstract

An intelligent system that supports real-time and offline feedback based on automated analysis of a trainee'"'"'s performance using data streams captured in the training process is disclosed.

Claims

1. A method of providing training comprising:

receiving at least one video stream from a video camera observing a trainee'"'"'s movements;

extract skill-related attributes from the at least one video stream to; and

displaying the video stream and the skill-related attributes.

2. The method of claim 1, wherein the skill-related attributes are displayed on a display in real-time.

3. The method of claim 1, further comprising:

receiving at least one data stream from a data glove; and

extracting skill-related attributes from the at least one data stream.

4. The method claim 3, further comprising

receiving at least one data stream from a motion tracker;

and

extract skill-related attributes from the at least one data stream.

5. The method of claim 1, wherein the extracted attributes comprise motion features in a region of interest.

6. The method of claim 5, wherein the motion features comprise spatial motion, radial motion, relative motion, angular motion and optic flow.

7. The method of claim 1, wherein the extracting step utilizes a random forest model.

8. An apparatus for training a trainee, comprising:

a laparoscopic surgery simulation system having a first camera and a video monitor;

a second camera for capturing a trainee'"'"'s hand movement; and

a computer for receiving video streams from the first and second cameras, the computer having a processor configured to extract skill-related attributes video streams to extract.

9. The apparatus of claim 8, further comprising kinematic sensors for capturing kinematics of the hands and fingers.

10. The apparatus of claim 9, wherein the kinematic sensor comprises a motion tracker.

11. The apparatus of claim 9, wherein the kinematic sensor comprises a data glove.

12. The apparatus of claim 9, wherein the skill-related attributes comprise smoothness of motion and acceleration.

13. A method of providing instructive feedback comprising:

decomposing a video sequence of a training procedure into primitive action units; and

rating each action unit using expressive attributes derived from established guidelines.

14. The method of claim 13, further comprising selecting an illustrative video as a reference from a pre-stored database.

15. The method of claim 13, further comprising storing a trainee'"'"'s practice sessions of the training procedure.

16. The method of claim 15, further comprising comparing different trainee practice sessions of the training procedure.

17. The method of claim 13, further comprising providing offline feedback.

18. The method of claim 13, further comprising providing live feedback.

19. The method of claim 13, wherein the expressive attributes are selected from the group comprising hands synchronization, instrument handling, suture handling, flow of operation and depth perception.

20. The method of claim 13, further comprising:

identifying worst action attributes of a trainee;

retrieving illustration video clips relating to the worst action attributes; and

presenting the illustration video clips to the trainee.