Methods and apparatus for displaying enhanced imaging data on a clinical image

• US 8,706,184 B2
• Filed: 10/07/2009
• Issued: 04/22/2014
• Est. Priority Date: 10/07/2009
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Mohr, Catherine, McDowall, Ian, Mohr, Paul
• Primary Examiner(s): Lamprecht, Joel
• Application Number:
  US12/575,093
• Publication Number:
  US 20110082369A1
• Time in Patent Office: 1,658 Days
• Field of Search: 600/407, 600/420, 600/426, 600/447, 600/160
• US Class Current: 600/407
• CPC Class Codes: A61B 1/000094 extracting biological structures
• CPC Class Codes: A61B 1/043 for fluorescence imaging
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/10064 Fluorescence image
• CPC Class Codes: G06T 2207/10068 Endoscopic image
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction

Abstract

In one embodiment of the invention, an apparatus includes a display device. The display device displays a desaturated image of tissue captured in the visible electro-magnetic (EM) spectrum from a body cavity; and a first color enhanced image combined with the desaturated image. The first color enhanced image represents the first data captured from the body cavity outside the visible electromagnetic spectrum. The relative brightness between the desaturated image and the first color enhanced image is set to emphasize the first data over the tissue captured in the visible electromagnetic spectrum to provide improved information content.

Claims

1. A method comprising:

desaturating, by a processor in a surgical system, a visible color image of tissue into a desaturated image, the visible color image being a reflected white light image captured from visible electro-magnetic spectrum light, the visible electro-magnetic spectrum light being from a surgical site in a patient;

coloring, by the processor, an enhanced image representing data captured from the surgical site with a visible color to form a color enhanced image, the data being captured from light outside the visible electro-magnetic spectrum;

superimposing, by the processor, the color enhanced image and the desaturated image together to form a blended image of the surgical site for display on a display device.

2. The method of claim 1, further comprising:

illuminating the surgical site the patient with visible light in the visible electro-magnetic spectrum;

capturing the visible color image of the surgical site in the patient; and

capturing the data from light from a biomarker in the surgical site.

3. The method of claim 2, further comprising:

prior to the capturing the data, exposing the surgical site in the patient to electro-magnetic radiation outside the visible electro-magnetic spectrum to excite the biomarker.

4. The method of claim 3, wherein the biomarker comprises indocyanine green;

and wherein the electro-magnetic radiation outside the visible electro-magnetic spectrum is near-infrared radiation.

5. The method of claim 1, further comprising:

prior to the superimposing, setting a first relative brightness between the color enhanced image and the desaturated image.

6. The method of claim 5, wherein

the first relative brightness is a predetermined brightness level, and

the desaturation of the visible color image is at a predetermined level of desaturation.

7. The method of claim 5, further comprising:

receiving an input from a user interface, the input indicating to vary the setting of the relative brightness between the color enhanced image and the desaturated image to a second relative brightness.

8. The method of claim 5, further comprising:

receiving an input from a user interface, the input indicating to vary the level of desaturation to desaturate the visible color image of the surgical site.

9. The method of claim 8, the input indicating to fully desaturate the visible color image of the surgical site, and thereby the desaturated image is a black and white image.

10. The method of claim 1, wherein the enhanced image is one of a near-infra-red fluorescence image, visible light fluorescence image, or a raster scan of spectral characteristics of the surgical site including clinical information with spatial variation.

11. The method of claim 1, wherein the enhanced image is formed of point measurements of a measureable tissue parameter.

12. The method of claim 11, wherein the measureable tissue parameter is tissue impedance, point detection of cancer cells, or point detection of a predetermined cell type.

13. The method of claim 1, further comprising temporally registering and spatially registering the color enhanced image with the desaturated image, prior to the superimposing.

14. An apparatus comprising:

an image processor, the image processor being configured to desaturate a captured visible color image into a desaturated image, the visible color image being a reflected white light image captured from visible electro-magnetic spectrum light, the visible electro-magnetic spectrum light being from a surgical site in a patient, the image processor being configured to color a first enhanced image representing first captured data with a first visible color to form a first enhanced color image, the first captured data being first data from the surgical site, and the first data being outside the visible electro-magnetic spectrum; and

a display device coupled to the image processor, the display device being configured to display a superpostion of the desaturated image and the color enhanced image together to form a blended image of the surgical site.

15. The apparatus of claim 14,

the image processor further being configured to color a second enhanced image representing second captured data with a second visible color to form a second enhanced color image, the second captured data being second data captured from the surgical site, the second data being outside the visible electro-magnetic spectrum, the first data being different from the second data, and the first color being different from the second color;

and

the display device further being configured to display a superimposition of the desaturated image, the first color enhanced image, and the second color enhanced image together to form the blended image of the surgical site.

16. The apparatus of claim 14, further comprising:

a user interface including a user selectable device to adjust the relative brightness between the desaturated image and the first color enhanced image displayed on the display device.

17. The apparatus of claim 15, further comprising:

a user interface including a first user selectable device to adjust the brightness of the desaturated image displayed on the display device and a second user selectable device to adjust the brightness of the second color enhanced image displayed on the display device.

18. The apparatus of claim 14, wherein

a camera unit coupled to the image processor, the camera unit comprising;

a first sensor sensitive to ranges of wavelengths in the visible electromagnetic spectrum; and

a second sensor sensitive to ranges of wavelengths outside the visible electromagnetic spectrum.

19. The apparatus of claim 14, wherein

a portion of the desaturated image includes a desaturated image of surgical tools.

20. The apparatus of claim 14, further comprising an illumination source configured to provide illumination that excites a biomarker in the surgical site wherein the first data is from the excited biomarker.

21. The apparatus of claim 20, wherein the biomarker comprises indocyanine green (ICG).

22. The apparatus of claim 14, wherein the first captured data is data captured from an X-ray of bone tissue under surface tissue.

23. A machine readable media have stored thereon instructions that when executed by a machine causes the machine to perform operations comprising:

reducing color in color images to form desaturated images, the color images being of tissue in a surgical site in a patient, and the color images being reflected white light images in the visible electro-magnetic spectrum;

making non-visible tissue feature images into color enhanced images, the tissue feature being in the surgical site; and

superimposing the color enhanced images and the desaturated images together to form blended images for display on at least one display device.

24. The machine readable media of claim 23, wherein the non-visible tissue feature images comprise images captured from light outside the visible electro-magnetic (EM) spectrum.

25. The machine readable media of claim 23, wherein

the non-visible characteristic tissue features of the surgical site are made visible over the tissue captured in desaturated images by a false color.

26. A method comprising:

capturing a visible color image of a surgical site illuminated by visible light, the visible color image being a reflected white light image captured from visible electro-magnetic spectrum light;

desaturating the visible color image of the surgical site into a visible gray image;

capturing a fluorescent image of fluorescing tissue in the surgical site, wherein the fluorescing tissue appears in a visible color; and

generating a displayable image comprising a superimposition of the visible fluorescent image and the visible gray image together to form a blended image of the surgical site.

27. The method of claim 26, wherein the desaturating comprises:

substantially removing all color from the visible color image to form the visible gray image.

28. The method of claim 26, wherein the obtaining of the fluorescent image comprises:

exposing the surgical site to electromagnetic radiation outside the visible electromagnetic spectrum.

29. The method of claim 26, further comprising:

coloring the fluorescing tissue in the fluorescent image with the visible color.

30. The method of claim 26, further comprising:

displaying the displayable image on a display device; and

adjusting a level of brightness of one or both of the visible gray image and the fluorescent image.

31. The method of claim 26, further comprising:

displaying the displayable image on a display device; and

adjusting a level of desaturation of the visible color image.