Method and apparatus for quantitative and depth resolved hyperspectral fluorescence and reflectance imaging for surgical guidance

• US 11,510,600 B2
• Filed: 02/15/2016
• Issued: 11/29/2022
• Est. Priority Date: 01/04/2012
• Status: Active Grant

• Current Assignee: The Trustees of Dartmounth College (Dartmouth College), University Health Network
• Original Assignee: The Trustees of Dartmounth College (Dartmouth College)
• Inventors: Valdes, Pablo, Leblond, Frederic, Paulsen, Keith D., Wilson, Brian Campbell, Roberts, David W., Jermyn, Michael
• Primary Examiner(s): Fernandez, Katherine L
• Application Number:
  US15/044,097
• Publication Number:
  US 20160278678A1
• Time in Patent Office: 2,479 Days
• CPC Class Codes: A61B 1/00009 of image signals during a use of endoscope
• CPC Class Codes: A61B 1/000094 extracting biological structures
• CPC Class Codes: A61B 1/00165 with light-conductive means, e.g. fibre optics A61B1/07 takes precedence
• CPC Class Codes: A61B 1/00193 adapted for stereoscopic vision
• CPC Class Codes: A61B 1/00194 adapted for three-dimensional imaging
• CPC Class Codes: A61B 1/042 characterised by a proximal camera, e.g. a CCD camera
• CPC Class Codes: A61B 1/043 for fluorescence imaging
• CPC Class Codes: A61B 1/05 characterised by the image sensor, e.g. camera, being in the distal end portion
• CPC Class Codes: A61B 1/0646 with illumination filters
• CPC Class Codes: A61B 1/0669 at proximal end of an endoscope
• CPC Class Codes: A61B 1/0684 using light emitting diodes [LED]
• CPC Class Codes: A61B 1/07 using light-conductive means, e.g. optical fibres
• CPC Class Codes: A61B 1/3132 for laparoscopy
• CPC Class Codes: A61B 5/0071 by measuring fluorescence emission
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/055 involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging
• CPC Class Codes: A61B 5/14546 for measuring analytes not otherwise provided for, e.g. ions, cytochromes
• CPC Class Codes: A61B 5/14553 specially adapted for cerebral tissue
• CPC Class Codes: A61B 5/14556 by fluorescence A61B5/14555 takes precedence
• CPC Class Codes: A61B 5/1459 invasive, e.g. introduced into the body by a catheter A61B5/1464 takes precedence
• CPC Class Codes: A61B 5/1495 : Calibrating or testing of in-vivo probes
• CPC Class Codes: A61B 5/4836 : Diagnosis combined with treatment in closed-loop systems or methods A61B5/0036 takes precedence
• CPC Class Codes: A61B 5/7264 : Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: G01J 2003/1269 : Electrooptic filter
• CPC Class Codes: G01J 2003/2826 : Multispectral imaging, e.g. filter imaging
• CPC Class Codes: G01J 3/0229 : using masks, aperture plates, spatial light modulators or spatial filters, e.g. reflective filters
• CPC Class Codes: G01J 3/1256 : using acousto-optic tunable filter; acousto-optic elements or systems G02F1/11, G02F1/33
• CPC Class Codes: G01J 3/2823 : Imaging spectrometer

Abstract

An imaging system, such as a surgical microscope, laparoscope, or endoscope or integrated with these devices, includes an illuminator providing patterned white light and/or fluorescent stimulus light. The system receives and images light hyperspectrally, in embodiments using a hyperspectral imaging array, and/or using narrowband tunable filters for passing filtered received light to an imager. Embodiments may construct a 3-D surface model from stereo images, and will estimate optical properties of the target using images taken in patterned light or using other approximations obtained from white light exposures. Hyperspectral images taken under stimulus light are displayed as fluorescent images, and corrected for optical properties of tissue to provide quantitative maps of fluorophore concentration. Spectral information from hyperspectral images is processed to provide depth of fluorophore below the tissue surface. Quantitative images of fluorescence at depth are also prepared. The images are displayed to a surgeon for use in surgery.

Claims

1. An optical and image processing system comprising:

a light source adapted to provide, in a first mode, light at a fluorescent stimulus wavelength;

a spatial modulator coupled to modulate light from the light source, forming spatially modulated fluorescent stimulus light, the spatial modulator configured to provide a plurality of different spatial light modulation patterns comprising light at the fluorescent stimulus wavelength, the spatial light modulation patterns each characterized by a spatial frequency;

projection apparatus configured to project onto tissue the spatially modulated fluorescent stimulus light, the tissue being illuminated with the plurality of different spatial light modulation patterns at the fluorescent stimulus wavelength, each spatial light pattern providing light varying in intensity across the tissue according to the spatial frequency;

a hyperspectral camera configured to receive light from the tissue and generate images therefrom; and

an image processing system coupled to the hyperspectral camera and configured with a memory containing machine readable instructions for performing depth resolved, fluorescent imaging by driving the spatial modulator to provide at least a first and a second predetermined spatial-light modulation pattern of the plurality of different spatial light modulation patterns, receiving first images generated by the hyperspectral camera at the fluorescent stimulus wavelength, at least one second image at a first fluorescent emissions wavelength while the tissue is illuminated with light of the fluorescent stimulus wavelength, and at least one third image at a second fluorescent emissions wavelength different from the first fluorescent emissions wavelength while the tissue is illuminated with the light of the fluorescent stimulus wavelength, for processing the received first, second, and third images generated by the hyperspectral camera to generate processed fluorescent images, and for displaying the processed fluorescent images;

wherein the first predetermined spatial light modulation pattern has a same spatial frequency as the second predetermined spatial light modulation pattern, and wherein the first predetermined spatial modulation pattern has spatial phase different from a spatial phase of the second predetermined spatial modulation pattern;

wherein the machine readable instructions for generating the processed fluorescent images comprise instructions for;

determining a relationship between depth and ratios of intensity in the second and third images for a fluorophore in tissue from images of the tissue as illuminated with the plurality of different spatial light modulation patterns;

applying stimulus wavelength light; and

determining a depth of the fluorophore at each pixel based upon the relationship between depth and the ratios of intensity in the second and third images;

where the hyperspectral camera is adapted to pass light received from the tissue through a filter device selected from the group consisting of a tiling pattern of optical filters on photosensors of an image sensor, the tiling pattern of optical filters comprising filters of at least 13 distinct wavelength bands between 400 nanometers and 1000 nanometers, and a tunable filter.

2. The system of claim 1 wherein the relationship between depth and ratios of intensity at the first and the second fluorescent emissions wavelength is determined on a per-pixel basis from the images of the tissue.

3. The system of claim 2 wherein the machine readable instructions further comprise instructions for determining optical properties of tissue, the optical properties of tissue including at least absorbance, and for correcting depth-resolved fluorescent images for the absorbance to provide images of absolute fluorophore concentration.

4. The system of claim 3 wherein the optical properties of tissue are determined at each voxel of a three-dimensional voxel-based model, and the three-dimensional voxel-based model is used in constructing the images of absolute fluorophore concentration.

5. The system of claim 1 wherein the light source is adapted to operate in a second mode providing light at fluorescent emissions wavelength and wherein the machine readable instructions for determining optical properties of tissue comprise instructions for:

determining reflectance and absorbance parameters at each pixel of an image at the fluorescent stimulus wavelength from the received first images, and determining absorbance parameters at each pixel of the image at the emissions wavelength from received images obtained while the light source is operated in the second mode and the tissue is illuminated with a plurality of spatial light patterns; and

using the reflectance and absorbance parameters at stimulus wavelength and the absorbance parameters at the emissions wavelength to correct the processed fluorescent images.

6. The system of claim 5 wherein the machine readable instructions include instructions for providing spatially modulated light when obtaining images from which the reflectance and absorbance parameters at stimulus and emissions wavelengths are determined.

7. An optical and image processing system of claim 6 wherein there are at least two cameras adapted to capture digital stereo images and the machine readable instructions further comprise instructions for extracting a surface profile from the stereo images.

8. An optical and image processing system of claim 7, wherein the machine readable instructions further comprise instructions for determining an intraoperative location of structures located in preoperative medical images, and for displaying the determined intraoperative location.

9. An optical and image processing system of claim 8 wherein the machine readable instructions further comprise instructions for displaying the determined intraoperative location with the processed fluorescent images.

10. An optical and image processing system of claim 7, wherein the machine readable instructions further comprise instructions for extracting a surface profile from depth-resolved fluorescent images.

11. An optical and image processing system of claim 7, further comprising a tracking subsystem adapted to determine a location and viewing angle of a display and wherein the machine readable instructions further comprise instructions for displaying rendered information selected from the group consisting of depth-resolved fluorescent images and intraoperative locations of structures as viewed from the determined location and viewing angle.

12. The system of claim 1 where the light source further comprises a broadband illumination source coupled to provide light through a tunable filter having a passband at the fluorescent stimulus wavelength, light from the tunable filter being provided to the spatial modulator;

and wherein the spatial modulator and projection apparatus is configurable to illuminate the tissue with the spatial light pattern in three different spatial phases.

13. The system of claim 1 where the hyperspectral camera is a wavelength-selective camera

where the light from the tissue passes through the tunable filter before reaching an imager.

14. The system of claim 1 wherein, during processing to generate the processed fluorescent images, the image processing system is configured to determine scattering and absorption parameters of the tissue at voxels of a three-dimensional model of the tissue and to use the scattering and absorption parameters to correct fluorescent emissions received from the hyperspectral camera at the fluorescent emissions wavelength.

15. An optical and image processing system comprising:

a light source adapted to provide, in a first mode, light at a fluorescent stimulus wavelength;

a spatial modulator coupled to modulate light from the light source, forming spatially modulated fluorescent stimulus light, the spatial modulator configured to provide a plurality of different spatial light modulation patterns comprising light at the fluorescent stimulus wavelength, the spatial light modulation patterns each characterized by a spatial frequency;

projection apparatus configured to project onto tissue the spatially modulated fluorescent stimulus light, the tissue being illuminated with the plurality of different spatial light modulation patterns at the fluorescent stimulus wavelength, each spatial light pattern providing light varying in intensity across the tissue according to the spatial frequency;

a hyperspectral camera configured to receive light from the tissue and generate images therefrom; and

an image processing system coupled to the hyperspectral camera and configured with a memory containing machine readable instructions for performing depth resolved, fluorescent imaging by driving the spatial modulator to provide at least a first and a second predetermined spatial-light modulation pattern of the plurality of different spatial light modulation patterns, receiving first images generated by the hyperspectral camera at the fluorescent stimulus wavelength, at least one second image at a first fluorescent emissions wavelength, and at least one third image at a second fluorescent emissions wavelength, for processing the received first, second, and third images generated by the hyperspectral camera to generate processed fluorescent images, and for displaying the processed fluorescent images;

wherein the first predetermined spatial light modulation pattern has a same spatial frequency as the second predetermined spatial light modulation pattern, and wherein the first predetermined spatial modulation pattern has spatial phase different from a spatial phase of the second predetermined spatial modulation pattern;

determining a relationship between depth and ratios of intensity in the second and third images for a fluorophore in tissue;

applying stimulus wavelength light;

determining a depth of the fluorophore at each pixel based upon the relationship between depth and the ratios of intensity in the second and third images; and

wherein the relationship between depth and ratios of intensity at the first and the second fluorescent emissions wavelengths are determined from images of the tissue;

wherein the machine readable instructions further comprise instructions for determining optical properties of tissue, the optical properties of tissue including at least absorbance, and for correcting depth-resolved fluorescent images for the absorbance to provide images of absolute fluorophore concentration;

wherein the optical properties of tissue are determined at each voxel of a three-dimensional voxel-based model, and the three-dimensional voxel-based model is used in constructing the images of absolute fluorophore concentration; and

wherein fluorophore concentration parameters of the three-dimensional voxel-based model are fit to at least the obtained second and third images, the fitting including simulating the three-dimensional voxel-based model using Monte Carlo based modeling of near-surface voxels and diffusion-based modeling of deep voxels.