Augmented stereoscopic visualization for a surgical robot using a captured fluorescence image and captured stereoscopic visible images

• US 8,228,368 B2
• Filed: 06/30/2008
• Issued: 07/24/2012
• Est. Priority Date: 04/26/2008
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Zhao, Wenyi, Mohr, Catherine J., Hasser, Christopher J., Hoffman, Brian D., Stern, John D., Zhao, Tao
• Primary Examiner(s): Dalencourt, Yves
• Application Number:
  US12/164,976
• Publication Number:
  US 20090268010A1
• Time in Patent Office: 1,485 Days
• Field of Search: 348/45, 348/51, 348/65, 600/109, 600/180, 600/181
• US Class Current: 348/45
• CPC Class Codes: A61B 1/000096 using artificial intelligence
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/00149 using articulated arms
• CPC Class Codes: A61B 1/00186 with imaging filters
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 1/00194 adapted for three-dimensional imaging
• CPC Class Codes: A61B 1/043 for fluorescence imaging
• CPC Class Codes: A61B 1/05 characterised by the image sensor, e.g. camera, being in the distal end portion
• CPC Class Codes: A61B 1/063 for monochromatic or narrow-band illumination
• CPC Class Codes: A61B 1/0638 providing two or more wavelengths
• CPC Class Codes: A61B 1/313 for introducing through surgical openings, e.g. laparoscopes
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2090/364 Correlation of different images or relation of image positions in respect to the body
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 5/0071 by measuring fluorescence emission
• CPC Class Codes: A61B 5/0075 by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy A61B5/0071 takes precedence
• CPC Class Codes: A61B 5/0084 for introduction into the body, e.g. by catheters A61B5/1459 takes precedence
• CPC Class Codes: A61B 90/30 Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G01N 21/6456 : Spatial resolved fluorescence measurements; Imaging
• CPC Class Codes: H04N 13/207 : using a single 2D image sensor
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence

Abstract

An illumination channel, a stereoscopic optical channel and another optical channel are held and positioned by a robotic surgical system. A first capture unit captures a stereoscopic visible image from the first light from the stereoscopic optical channel while a second capture unit captures a fluorescence image from the second light from the other optical channel. An intelligent image processing system receives the captured stereoscopic visible image and the captured fluorescence image and generates a stereoscopic pair of fluorescence images. An augmented stereoscopic display system outputs a real-time stereoscopic image comprising a three-dimensional presentation of a blend of the stereoscopic visible image and the stereoscopic pair of fluorescence images.

Claims

1. A robotic surgical system comprising:

an illumination channel, held and positioned by the robotic surgical system, wherein light from the illumination channel illuminates tissue;

a stereoscopic optical channel, held and positioned by the robotic surgical system, configured to transport first light from the tissue;

another optical channel, held and positioned by the robotic surgical system, configured to transport second light from the tissue wherein the stereoscopic optical channel is different from the another optical channel;

an image capture system comprising;

a first capture unit coupled to the stereoscopic optical channel wherein the first capture unit captures a stereoscopic visible image from the first light; and

a second capture unit coupled to the another optical channel wherein the second capture unit captures a fluorescence image from the second light;

an image processing system coupled to the first capture unit and to the second capture unit to receive the stereoscopic visible image and to receive the fluorescence image,

wherein the image processing system is configured to use information associated with the fluorescence image to generate a first fluorescence image and to generate a second fluorescence image, and

wherein the first fluorescence image and the second fluorescence image comprise a stereoscopic pair of fluorescence images; and

an augmented stereoscopic display system coupled to the intelligent image processing system and to the image capture system, wherein the augmented stereoscopic display system outputs a stereoscopic image comprising a blend of the stereoscopic visible image and the stereoscopic pair of fluorescence images;

wherein the augmented stereoscopic display system further comprises;

a first blend circuit coupled to receive one of the first and second fluorescence images in the stereoscopic pair of fluorescence images, and to receive a visible left image of the stereoscopic visible image;

a second blend circuit coupled to receive another one of the first and second fluorescence images in the stereoscopic pair of fluorescence images, and to receive a visible right image of the stereoscopic visible image; and

a stereoscopic display coupled to the first blend circuit and to the second blend circuit.

2. The robotic surgical system of claim 1 wherein the illumination channel and the stereoscopic optical channel are in a first endoscope and the another optical channel is in a second endoscope different from the first endoscope.

3. The robotic surgical system of claim 2 further comprising:

another illumination channel in the second endoscope.

4. The robotic surgical system of claim 1 wherein the image processing system further comprises:

a single-step calibration wherein the single-step calibration determines a displacement necessary to bring images from the stereoscopic optical channel and the another optical channel into alignment.

5. The robotic surgical system of claim 1 wherein the image processing system further comprises:

a spatial image registration configured to register the fluorescence image to a visible left image in the stereoscopic visible image and to a visible right image in the stereoscopic visible image.

6. The robotic surgical system of claim 1 wherein the image processing system further comprises:

an image warper configured to convert the fluorescence image into the first and second fluorescence images in the stereoscopic pair of fluorescence images.

7. The robotic surgical system of claim 1 wherein the augmented stereoscopic display system further comprises:

a display mode select, coupled to the first and second blend circuits, having a plurality of selectable modes,

wherein upon selection of a first mode, the stereoscopic display provides a blend of the stereoscopic visible image and the stereoscopic pair of fluorescence images, and

wherein upon selection of a second mode, the stereoscopic display provides only the stereoscopic visible image.

8. The robotic surgical system of claim 1 further comprising:

a combination light source, coupled to the illumination channel, comprising;

a first light source; and

a second light source, wherein illumination light from the second light source is different from illumination light from the first light source.

9. A robotic surgical system comprising:

an illumination channel, held and positioned by the robotic surgical system, wherein light from the illumination channel illuminates tissue;

a stereoscopic optical channel, held and positioned by the robotic surgical system, configured to transport first light from the tissue;

another optical channel, held and positioned by the robotic surgical system, configured to transport second light from the tissue wherein the stereoscopic optical channel is different from the another optical channel;

a combination light source, coupled to the illumination channel, comprising;

a first light source; and

a second light source wherein illumination light from the second light source is different from illumination light from the first light source;

an image capture system comprising;

a first capture unit coupled to the stereoscopic optical channel wherein the first capture unit captures a stereoscopic visible image from the first light; and

a second capture unit coupled to the another optical channel wherein the second capture unit captures a fluorescence image from the second light;

an image processing system, coupled to the first capture unit and to the second capture unit to receive the stereoscopic visible image and to receive the fluorescence image, comprising;

a single-step calibration wherein the single-step calibration determines a displacement necessary to bring images from the stereoscopic optical channel and from the another optical channel into alignment;

a spatial image registration, coupled to the single-step calibration, for registering the fluorescence image to a visible left image in the stereoscopic visible image and to a visible right image in the stereoscopic visible image;

an image warper coupled to the spatial image registration and coupled to receive the fluorescence image, wherein the image warper is configured to convert the fluorescence image into a first fluorescence image, and to convert the fluorescence image into a second fluorescence image, and wherein the first fluorescence image and the second fluorescence image comprise a stereoscopic pair of fluorescence images; and

an augmented stereoscopic display system coupled to the image processing system and to the image capture system, wherein the augmented stereoscopic display system outputs a stereoscopic image comprising a blend of the stereoscopic visible image and the stereoscopic pair of fluorescence images and further wherein the augmented stereoscopic display system comprises;

a first blend circuit coupled to receive one of the first and second fluorescence images in the stereoscopic pair of fluorescence images, and to receive a visible left image of the stereoscopic visible image;

a second blend circuit coupled to receive an other of the first and second fluorescence images in the stereoscopic pair of fluorescence images, and to receive a visible right image of the stereoscopic visible image;

a stereoscopic display coupled to the first blend circuit and to the second blend circuit; and

a display mode select, coupled to the first and second blend circuits, having a plurality of selectable modes,

wherein upon selection of a first mode, the stereoscopic display provides a blend of the stereoscopic visible image and the stereoscopic pair of fluorescence images, and

wherein upon selection of a second mode, the stereoscopic display provides only the real time stereoscopic visible image.

10. A method comprising:

capturing a stereoscopic visible image of tissue from a stereoscopic optical channel held and positioned by a robotic surgical system;

capturing a fluorescence image of the tissue from another optical channel held and positioned by the robotic surgical system, wherein the stereoscopic optical channel is different from the another optical channel;

processing information associated with the fluorescence image to generate a first fluorescence image, and to generate a second fluorescence image, wherein the first fluorescence image and the second fluorescence image comprise a stereoscopic pair of fluorescence images; and

generating an augmented stereoscopic image of the tissue comprising a blend of the stereoscopic visible image and the stereoscopic pair of fluorescence images;

wherein the processing the information associated with the fluorescence image further comprises;

converting the information associated with the fluorescence image into the first and second fluorescence images using an image warper; and

wherein the generating an augmented stereoscopic image further comprises;

blending one of the first and second fluorescence images in the stereoscopic pair of fluorescence images and a visible left image of the stereoscopic visible image; and

blending another one of the first and second fluorescence images in the stereoscopic pair of fluorescence images and a visible right image of the stereoscopic visible image.

11. The method claim 10 wherein the illumination channel and the stereoscopic optical channel are in a first endoscope and the another optical channel is in a second endoscope different from the first endoscope.

12. The method of claim 10 wherein the processing the information associated with the fluorescence image further comprises:

determining a displacement necessary to bring images from the stereoscopic optical channel and the another optical channel into alignment.

13. The method of claim 10 wherein the processing the information associated with the fluorescence image further comprises:

spatially registering the fluorescence image to a visible left image in the stereoscopic visible image and to a visible right image in the stereoscopic visible image.

14. The method claim 10 further comprising:

performing the generating an augmented stereoscopic image of the tissue comprising a blend of the stereoscopic visible image and the stereoscopic pair of fluorescence images upon a surgeon selecting a first display mode; and

generating only the stereoscopic visible image, upon the surgeon selecting a second display mode.

15. The method of claim 11 further comprising:

providing light from a combination light source to an illumination path in the first endoscope.

16. The method of claim 11 further comprising:

providing light from a first light source in a combination light source to an illumination path in the first endoscope; and

providing light from a second light source in the combination light source to an illumination path in the second endoscope.