System and method for augmented reality navigation in a medical intervention procedure

• US 7,774,044 B2
• Filed: 02/11/2005
• Issued: 08/10/2010
• Est. Priority Date: 02/17/2004
• Status: Active Grant

• Current Assignee: Siemens Medical Solutions USA Incorporated (Siemens AG)
• Original Assignee: Siemens Medical Solutions USA Incorporated (Siemens AG)
• Inventors: Vogt, Sebastian, Sauer, Frank, Khamene, Ali
• Primary Examiner(s): Winakur; Eric F
• Assistant Examiner(s): Laryea; Lawrence N
• Application Number:
  US11/056,317
• Publication Number:
  US 20050203380A1
• Time in Patent Office: 2,006 Days
• Field of Search: 600/426, 600/427, 600/429, 600/439, 600/417, 600/424, 600/411, 354/126, 359/850, 359/462, 359/630, 359/51, 345 8- 9, 382/128, 382130-133
• US Class Current: 600/424
• CPC Class Codes: A61B 2034/107 Visualisation of planned trajectories or target regions
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/2072 Reference field transducer attached to an instrument or patient
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 90/20 Surgical microscopes characterised by non-optical aspects
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 7/002 Mounting on the human body
• CPC Class Codes: G06F 3/012 Head tracking input arrangements

Abstract

A method for augmented reality navigation of a medical intervention includes providing a stereoscopic head mounted display, the display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance display. During a medical intervention on a patient, the patient'"'"'s body pose is determined from a rigid body transformation between the tracking camera and frame markers on the scanning table, and the pose of an intervention instrument with respect to the table is determined. A visual representation of the patient overlaid with an image of the intervention target, the instrument, and a path for guiding the instrument to perform said medical intervention is displayed in the stereoscopic guidance display.

Claims

1. A method for augmented reality navigation during a medical intervention, said method comprising the steps of:

providing stereoscopic head mounted display, said display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance display;

providing a plurality of markers on a frame attached to a table;

providing a medical instrument for performing said medical intervention, and providing a plurality of markers on the instrument;

during a medical intervention on a patient positioned on said table, determining a rigid body transformation between the tracking camera and the frame markers, and determining said patient'"'"'s body pose from said rigid body transformation;

determining the pose of said medical instrument with respect to the table from the patient'"'"'s body pose and said instrument markers;

displaying in the stereoscopic guidance display a visual representation of said patient, said instrument, and a path for guiding said instrument to perform said medical intervention, wherein the visual representation of said patient and said instrument is generated based on a signal received from said viewing cameras, and wherein the visual representation of said patient is overlaid with a medical image of a target of said intervention; and

color-coding the visual representation of the medical instrument to indicate penetration of said medical instrument toward said target in real-time to show the time evolution of the medical instrument'"'"'s trajectory.

2. The method of claim 1, further comprising the step of, prior to the medical intervention, determining a rigid body transformation between the tracking camera and the viewing cameras.

3. The method of claim 1, further comprising the step of, prior to the medical intervention, determining a relationship between the medical image and the tracking camera.

4. The method of claim 3, further comprising providing a calibration phantom for determining a relationship between the medical image and the tracking camera, wherein said calibration phantom includes magnetic and optical markers.

5. The method of claim 1, further comprising monitoring the patient'"'"'s respiration during said intervention, and altering said visual representation of said patient in accordance with said respiration.

6. The method of claim 1, wherein said image of said target is a magnetic resonance image.

7. The method of claim 1, wherein said instrument is a biopsy needle.

8. The method of claim 1, wherein the tracking camera is sensitive to the near-infrared spectrum, and comprises a wide angle lens and a ring shaped infrared flash.

9. The method of claim 1, further comprising a plurality of tracking cameras for positioning the head mounted display.

10. A program storage device readable by a computer, tangibly embodying a program of instructions executable by the computer to perform the method steps for augmented reality navigation during a medical intervention, said method comprising the steps of:

determining a rigid body transformation between a tracking camera and a plurality of frame markers attached to a table;

determining a patient'"'"'s body pose from said rigid body transformation;

during a medical intervention on said patient, wherein said patient is positioned on said table, determining the pose of a medical instrument with respect to the table from the patient'"'"'s body pose and a plurality of instrument markers;

providing a stereoscopic video-view of the patient in real-time in a stereoscopic head mounted display, said display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance display;

displaying in the stereoscopic video-view a visual representation of said instrument, and a path for guiding said instrument to perform said medical intervention, wherein the visual representation of said patient is overlaid with an image of a target of said intervention; and

color-coding the visual representation of the medical instrument to indicate penetration of said medical instrument toward said target in real-time to show the time evolution of the medical instrument'"'"'s trajectory.

11. The computer readable program storage device of claim 10, wherein the tracking camera is sensitive to the near-infrared spectrum, and comprises a wide angle lens and a ring shaped infrared flash.

12. The computer readable program storage device of claim 10, wherein the method further comprises the step of, prior to the medical intervention, determining a rigid body transformation between the tracking camera and a pair of stereo viewing cameras.

13. The computer readable program storage device of claim 10, wherein the method further comprises the step of, prior to the medical intervention, determining a relationship between the medical image and the tracking camera.

14. The computer readable program storage device of claim 13, wherein the method further comprises the step of providing a calibration phantom, said calibration phantom including magnetic and optical markers, wherein said calibration phantom is used to determine the relationship between the medical image and the tracking camera.

15. The computer readable program storage device of claim 10, wherein said method further comprises monitoring the patient'"'"'s respiration during said intervention, and altering said visual representation of said patient in accordance with said respiration.

16. The computer readable program storage device of claim 10, wherein said image of said target is a magnetic resonance image.

17. The computer readable program storage device of claim 10, wherein said instrument is a biopsy needle.

18. A method for augmented reality navigation during a magnetic resonance guided intervention, said method comprising the steps of:

providing stereoscopic head mounted display, said display including a pair of stereo viewing cameras, at least one tracking camera, and a stereoscopic guidance display;

providing a biopsy needle for performing said intervention;

providing a scanning table having a set of attached markers;

providing a calibration phantom that includes magnetic and optical markers;

determining a transformation between said at least one tracking camera and said scanning table by scanning said calibration phantom;

during a medical intervention on a patient positioned on said table, determining said patient'"'"'s body pose from a rigid body transformation between the tracking camera and the table;

determining the pose of said biopsy needle with respect to the table from the patient'"'"'s body pose;

displaying in the stereoscopic guidance display a visual representation of said patient, said needle, and a path for guiding said needle to perform said intervention, wherein the visual representation of said patient and said needle is generated based on a signal received from said viewing cameras, and wherein the visual representation of said patient is overlaid with a magnetic resonance image of a target of said intervention; and

displaying concentric circles in the guidance display to indicate distance of the needle to the target.

19. The method of claim 18, further comprising the step of, prior to said intervention, determining a rigid body transformation between the tracking camera and the viewing cameras.

20. The method of claim 18, wherein the tracking camera is sensitive to the near-infrared spectrum, and comprises a wide angle lens and a ring shaped infrared flash.

21. The method of claim 18, wherein said needle is provided with a plurality of markers, and wherein said markers are used in determining the pose of said needle.

22. The method of claim 18, further comprising providing a frame on said table, and providing a plurality of markers on said frame, wherein said markers are used to determine the rigid body transformation between said tracking camera and the patient'"'"'s body.