Method and system for simulating medical procedures including virtual reality and control method and system for use therein

• US 5,769,640 A
• Filed: 08/10/1995
• Issued: 06/23/1998
• Est. Priority Date: 12/02/1992
• Status: Expired due to Term

• Current Assignee: Immersion Corporation
• Original Assignee: Cybernet Systems Corporation
• Inventors: Griffin, Jennifer Lynn, Jacobus, Charles J.
• Primary Examiner(s): Apley, Richard J.
• Assistant Examiner(s): Rovnak, John Edmund
• Application Number:
  US08/513,488
• Time in Patent Office: 1,048 Days
• Field of Search: 434/262, 434/267, 434/268, 434/272
• US Class Current: 434/262
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G05B 2219/35416 3-D joystick
• CPC Class Codes: G05B 2219/37396 Tactile feedback, operator feels reaction, force reflection
• CPC Class Codes: G06F 2203/015 Force feedback applied to a joystick
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G09B 23/285 for injections, endoscopy, bronchoscopy, sigmoidscopy, insertion of contraceptive devices or enemas

Abstract

A method and system for simulating medical procedures is presented. During an actual medical procedure, the actions of a medical instrument are measured and recorded. These actions are generated on a member representative of the medical instrument to simulate the actual medical procedure. Forces acting upon the medical instrument are sensed and simulated in the member using a force/tactile reflecting mechanism. The preferred embodiment uses virtual reality technology including image processing, three-dimensional graphics and display methods, simulated force/tactile reflection, head/hand movement, position sensing, and sound generation to provide an accurate simulation of endoscopic medical procedures.

Claims

1. A method of providing a virtual reality in response to a position and orientation of a member representative of a medical instrument, the method comprising the steps of:

providing a first medical instrument having sensors for position and pressure;

performing an actual medical procedure using the first instrument;

measuring and recording during the actual medical procedure data representative of the position and orientation of the first instrument and the pressures experienced by the first instrument;

providing a member representative of the first medical instrument;

generating an electrical signal for each of a plurality of degrees of freedom of the member as a function of the position and orientation of the member in three-dimensional space;

generating at least one virtual reality force field in response to the generated electric signals as a function a the recorded position and pressures experienced by the first instrument;

generating a force signal for each degree of freedom as a function of the force field; and

directing a force on the member for each force signal, the generated forces providing the virtual reality.

2. The method of claim 1 wherein the actual medical procedure includes a pre-medical diagnostic procedure.

3. The method of claim 1 wherein the generated forces are tactile forces.

4. The method of claim 1, further including the steps of:

providing a second instrument including a sensor for gathering images of the actual medical procedure;

storing data representative of the images in a time-coordinated manner along with the recorded position, orientation and pressure data; and

replaying the images in a time-coordinated manner in conjunction with the direction of the forces on the member, the combination of the generated forces and the images providing the virtual reality.

5. The method of claim 1, further including the steps of:

storing data representative of images obtained from a previously obtained medical diagnostic procedure; and

replaying the images in conjunction with the direction of the forces on the member, the combination of the generated forces and the images providing the virtual reality.

6. The method of claim 5, wherein the data representative of images obtained from a previously obtained medical diagnostic procedure includes CT data.

7. The method of claim 5, wherein the data representative of images obtained from a previously obtained medical diagnostic procedure includes PET data.

8. The method of claim 5, wherein the data representative of images obtained from a previously obtained medical diagnostic procedure includes MRI data.

9. The method of claim 4, wherein the step of storing data representative of the images includes storing such data at regular intervals, and interpolating the data between the intervals during the replaying of the images.

10. The method of claim 1, wherein the step of measuring and recording the position, orientation and pressure data includes measuring and recording such data at regular intervals, the method further including the step of interpolating the data between the intervals during the generation of the virtual reality force field.

11. A system for providing a virtual reality in response to a position and orientation of a member representative of a medical instrument, the system comprising:

a first medical instrument including a plurality of sensors outputting electrical signals representative of the a position and orientation of the instrument, and the pressure experienced by the instrument, during use in an actual medical procedure;

a digitizer for converting the electrical signals received from the first instrument into a plurality of position, orientation, and pressure data sets;

a recorder for storing the data sets;

a member including a tactile/force reflecting mechanism responsive to the position and orientation of the member;

a programmed controller interfaced to the tactile/force reflecting mechanism for providing force feedback to the member; and

a virtual reality control computer in operative communication with the recorder and programmed controller, the computer being operative to map the position and orientation of the member into the stored data sets and determine any interactions therebetween, thereby enabling a user to experience the position and orientation of the instrument, and the pressure experienced by the instrument, through force feedback provided to the member.

12. The system of claim 11, further including a second medical instrument including:

a second instrument including a sensor for gathering images of the actual medical procedure; and

wherein the virtual reality control computer is further operative to map the images into the stored data sets, thereby enabling the user to view the position and orientation of the instrument while experiencing the force feedback provided to the member.

13. The system of claim 12, further including a three-dimensional display system.

14. The system of claim 12, further including an image warp processor operative to interpolate between two of the images through corresponding polygon patches.

15. The system of claim 12, further including means for overlaying imagery previously obtained from a medical diagnostic procedure.

16. The system of claim 11, further including a time base for providing time stamps associated with instrument locations, forces and pressures.

17. The system of claim 11, further including playback apparatus incorporating a member representative of the first medical instrument and a direct interface to the recorder, thereby providing immediate feedback of the actual medical procedure to a trainee.

18. The system of claim 11, further including means for queuing the first instrument into the virtual reality.

19. A method of providing a virtual reality in response to a position and orientation of a member representative of a medical instrument, the method comprising the steps of:

storing a geometrical model of an organ, including the size, position, and one or more surface characteristics associated therewith,

providing a member representative of a medical instrument;

generating an electrical signal for each of a plurality of degrees of freedom of the member as a function of the position and orientation of the member in three-dimensional space;

generating at least one virtual reality force field in response to the generated electric signals as a function of the size, position, and surface characteristics of the organ;

generating a force signal for each degree of freedom as a function of the force field; and

directing a force on the member for each force signal, the generated forces providing the virtual reality.

20. The method of claim 19, wherein one of the surface characteristics is elasticity.

21. The method of claim 19, wherein one of the surface characteristics is resistance to shearing.

22. The method of claim 19, wherein one of the surface characteristics is slipperiness.