System and method for high precision multi-aperture spectral imaging

• US 10,740,884 B2
• Filed: 01/09/2020
• Issued: 08/11/2020
• Est. Priority Date: 12/14/2018
• Status: Active Grant

• Current Assignee: Spectral Md Inc.
• Original Assignee: Spectral Md Inc.
• Inventors: McCall, Brian, Fan, Wensheng, Dwight, Jason, Gao, Zhicun, Thatcher, Jeffrey E., DiMaio, John Michael
• Primary Examiner(s): Adams, Eileen M
• Application Number:
  US16/738,910
• Publication Number:
  US 20200193580A1
• Time in Patent Office: 215 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10036 Multispectral image; Hyperspectral image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Generally described, one or more aspects of the present application correspond to systems and techniques for spectral imaging using a multi-aperture system with curved multi-bandpass filters positioned over each aperture. The present disclosure further relates to techniques for implementing spectral unmixing and image registration to generate a spectral datacube using image information received from such imaging systems. Aspects of the present disclosure relate to using such a datacube to analyze the imaged object, for example to analyze tissue in a clinical setting, perform biometric recognition, or perform materials analysis.

Claims

1. A multispectral image system, comprising:

at least one image sensor;

a first aperture positioned to allow incoming light to pass to a first sensor region of the at least one image sensor;

a second aperture positioned to allow incoming light to pass to a second sensor region of the at least one image sensor;

a first multi-bandpass filter positioned over the first aperture, wherein the first multi-bandpass filter is configured to allow passage of light in at least a common waveband and a first unique waveband;

a second multi-bandpass filter positioned over the second aperture, wherein the second multi-bandpass filter is configured to allow passage of light in at least the common waveband and a second unique waveband;

a memory storing instructions for generating a multispectral datacube; and

at least one processor configured by the instructions to at least;

receive signals from the first and second sensor regions; and

process the signals to generate the multispectral datacube, wherein the multispectral datacube includes a first spectral channel corresponding to the common waveband, a second spectral channel corresponding to the first unique waveband, and a third spectral channel corresponding to the second unique waveband;

wherein processing the signals comprises at least;

estimating, using information from the first spectral channel corresponding to the common waveband, a disparity between first image data captured by the first sensor region and second image data captured by the second sensor region; and

using the disparity to align the image data of the first, second, and third spectral channels to generate the multispectral datacube.

2. The multispectral image system of claim 1, wherein the at least one processor is configured by the instructions to at least perform spectral unmixing to generate image data for each of the first, second, and third spectral channels.

3. The multispectral image system of claim 1, further comprising at least one light source configured to emit light at the common waveband, the first unique waveband, and the second unique waveband.

4. The multispectral image system of claim 3, further comprising a diffusing optical element positioned over the at least one light source.

5. The multispectral image system of claim 1, wherein at least the first multi-bandpass filter is a curved filter.

6. A method of using the multispectral image system of claim 1 to generate the multispectral datacube, comprising:

capturing a first exposure comprising the first, second, and third spectral channels;

capturing a second exposure comprising the first, second, and third spectral channels; and

generating the multispectral datacube based at least partly on using the second exposure to reduce a signal to noise ratio of the first exposure.

7. The method of claim 6, further comprising registering images captured by the first and second sensor regions to one another based on information in the first spectral channel.

8. The method of claim 7, further comprising generating the multispectral datacube based on a result of registering the images captured by the first and second sensor regions to one another.

9. The method of claim 7, wherein the images represent information captured in the first spectral channel by the first and second sensor regions, the method further comprising:

identifying a disparity between the images captured by the first and second sensor regions in the first spectral channel corresponding to the common waveband;

using the disparity to register a first additional image to the images, the first additional image representing information captured by the first sensor region in the second spectral channel; and

using the disparity to register a second additional image to the images, the second additional image representing information captured by the second sensor region in the third spectral channel.

10. The method of claim 9, wherein each exposure comprises more than three spectral channels, the method further comprising using the disparity to register at least a third additional image to the images, the third additional image representing information captured by a third sensor region in the fourth spectral channel, wherein a total number of additional images registered to the images is one less than the total number of spectral channels.

11. The method of claim 9, further comprising using the disparity to determine at least one of topographic information or depth information.

12. The method of claim 11, wherein the disparity is used to determine depth information for individual pixels.

13. The method of claim 12, further comprising using the depth information to determine a volume of an imaged wound.

14. The method of claim 7, wherein the multispectral image system further comprises an illuminant configured to project one or more points, fringes, grids, random speckle, or other spatial pattern, the method further comprising:

projecting, by the illuminant into a field of view of at least the first aperture, light comprising one or more points, fringes, grids, random speckle, or other spatial pattern; and

capturing an additional exposure using the light projected by the illuminant;

wherein the registering is based at least in part on the additional exposure.

15. The method of claim 14, wherein the light projected by the illuminant comprises at least one of light having a wavelength in the common waveband, broadband illumination, or cumulatively visible illumination.

16. The method of claim 14, wherein the additional exposure is used to confirm an accuracy of the registration.

17. The method of claim 14, wherein the additional exposure is included in a calculation of the disparity to improve an accuracy of the registration.

18. The method of claim 14, wherein the illuminant projects a plurality of points, fringes, grids, speckle, or other spatial patterns in unique configurations in at least the common waveband, broadband illumination, or cumulatively visible illumination.

19. The method of claim 7, wherein the multispectral image system further comprises an additional sensor configured to detect the shape of objects in the field of view of the at least one imaging sensor, and wherein the registering is based at least in part on a shape detected by the additional sensor.

20. The method of claim 7, wherein the illuminant projects the light into a field of view of a plurality of apertures of the multispectral image system.