Method and system for hyperspectral light field imaging

• US 10,641,658 B1
• Filed: 11/08/2019
• Issued: 05/05/2020
• Est. Priority Date: 06/05/2017
• Status: Active Grant

• Current Assignee: Shanghaitech University
• Original Assignee: Shanghaitech University
• Inventors: Yu, Jingyi
• Primary Examiner(s): Perlman, David
• Application Number:
  US16/678,104
• Publication Number:
  US 20200141804A1
• Time in Patent Office: 179 Days
• Field of Search: None
• CPC Class Codes: G01J 2003/2826 Multispectral imaging, e.g. filter imaging
• CPC Class Codes: G01J 3/2823 Imaging spectrometer
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10036 Multispectral image; Hyperspectral image
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: G06V 20/194 using hyperspectral data, i.e. more or other wavelengths than RGB
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

A method for generating hyperspectral data-cubes based on a plurality of hyperspectral light field (H-LF) images is disclosed. Each H-LF image may have a different view and a different spectral band. The method may include calculating a magnitude histogram, a direction histogram, and an overlapping histogram of oriented gradient for a plurality of pixels; developing a spectral-invariant feature descriptor by combining the magnitude histogram, the direction histogram, and the overlapping histogram of oriented gradient; obtaining a correspondence cost of the H-LF images based on the spectral-invariable feature descriptor; performing H-LF stereo matching on the H-LF images to obtain a disparity map of a reference view; and generating hyperspectral data-cubes by using the disparity map of the reference view. A bin in the overlapping histogram of oriented gradient may comprise overlapping ranges of directions.

Claims

1. A method of generating hyperspectral data-cubes based on a plurality of hyperspectral light field (H-LF) images, wherein each H-LF image comprises a different view and a different spectral band, the method comprising:

calculating a magnitude histogram, a direction histogram, and an overlapping histogram of oriented gradient for a plurality of pixels, wherein a bin in the overlapping histogram of oriented gradient comprises overlapping ranges of directions;

developing a spectral-invariant feature descriptor by combining the magnitude histogram, the direction histogram, and the overlapping histogram of oriented gradient;

obtaining a correspondence cost of the H-LF images based on the spectral-invariable feature descriptor;

performing H-LF stereo matching on the H-LF images to obtain a disparity map of a reference view; and

generating hyperspectral data-cubes by using the disparity map of the reference view.

2. The method of claim 1, wherein the spectral-invariant feature descriptor measures both edge and non-edge features.

3. The method of claim 2, wherein the non-edge feature is described by coupling the magnitude histogram and the direction histogram, and the edge feature is described by the overlapping histogram of oriented gradient.

4. The method of claim 3, further comprising:

combining the magnitude histogram, the direction histogram, and the overlapping histogram of oriented gradient using a weight matrix and a pyramid structure to form the spectral invariant feature descriptor.

5. The method of claim 1, further comprising:

comparing the spectral-invariant feature descriptor to develop a similarity metric using a bidirectional weighted normalized cross correlation (BWNCC).

6. The method of claim 1, further comprising:

obtaining an estimated spectra by synthesizing RGB color values from the H-LF images and mapping the RGB color values to a spectral band; and

obtaining a defocus cost of the H-LF images based on a comparison between estimated spectra and captured spectra of the H-LF images.

7. The method of claim 6, wherein a Kullback-Leibler divergence is used in the comparison between the estimated spectra and the captured spectra of the H-LF images.

8. The method of claim 6, further comprising:

performing H-LF stereo matching on the H-LF images to obtain the disparity map of the reference view based on the correspondence cost and the defocus cost.

9. The method of claim 1, further comprising:

estimating an initial disparity map of the reference view based on the correspondence cost by treating occluding and non-occluding pixels separately.

10. The method of claim 1, further comprising:

capturing the plurality of H-LF images by a plurality of multi-view hyperspectral data terminals placed in a rectangular array comprising a plurality of rows and a plurality of columns, wherein each terminal consists a monochrome camera and a narrow bandpass optical filter.

11. A method of generating hyperspectral data-cubes based on a plurality of hyperspectral light field (H-LF) images, wherein each H-LF image comprises a different view and a different spectral band, the method comprising:

obtaining an estimated spectra by synthesizing RGB color values from the H-LF images and mapping the RGB color values to a spectral band;

obtaining a defocus cost of the H-LF images based on a comparison between estimated spectra and captured spectra of the H-LF images;

performing H-LF stereo matching on the H-LF images to obtain a disparity map of a reference view; and

generating hyperspectral data-cubes by using the disparity map of the reference view.

12. The method of claim 11, wherein a Kullback-Leibler divergence is used in the comparison between the estimated spectra and the captured spectra of the H-LF images.

13. The method of claim 12, further comprising:

performing H-LF stereo matching on the H-LF images to obtain the disparity map of the reference view based on the defocus cost.

14. The method of claim 13, further comprising:

calculating a magnitude histogram, a direction histogram, and an overlapping histogram of oriented gradient for the plurality of pixels, wherein a bin in the overlapping histogram of oriented gradient comprises overlapping ranges of directions;

developing a spectral-invariant feature descriptor by combining the magnitude histogram, the direction histogram, and the overlapping histogram of oriented gradient; and

obtaining a correspondence cost of the H-LF images based on the spectral-invariable feature descriptor.

15. The method of claim 14, wherein the spectral-invariant feature descriptor measures both edge and non-edge features.

16. The method of claim 15, wherein the non-edge feature is described by coupling the magnitude histogram and the direction histogram, and the edge feature is described by the overlapping histogram of oriented gradient.

17. The method of claim 16, further comprising:

combining the magnitude histogram, the direction histogram, and the overlapping histogram of oriented gradient using a weight matrix and a pyramid structure to form the spectral invariant feature descriptor.

18. The method of claim 13, further comprising:

comparing the spectral-invariant feature descriptor to develop a similarity metric using a bidirectional weighted normalized cross correlation (BWNCC).

19. The method of claim 11, further comprising:

estimating an initial disparity map of the reference view based on the correspondence cost by treating occluding and non-occluding pixels separately.

20. The method of claim 11, further comprising:

capturing the plurality of H-LF images by a plurality of multi-view hyperspectral data terminals placed in a rectangular array comprising a plurality of rows and a plurality of columns, wherein each terminal consists a monochrome camera and a narrow bandpass optical filter.

21. A hyperspectral light field imaging (HLFI) system for capturing hyperspectral light field (H-LF) images to generate hyperspectral data-cubes, the system comprising:

a plurality of multi-view hyperspectral data terminals configured to capture a plurality of H-LF images, wherein each H-LF image comprises a different view and a different spectral band; and

a data processing unit configured to;

calculate a magnitude histogram, a direction histogram, and an overlapping histogram of oriented gradient for a plurality of pixels, wherein a bin in the overlapping histogram of oriented gradient comprises overlapping ranges of directions;

develop a spectral-invariant feature descriptor by combining the magnitude histogram, the direction histogram, and the overlapping histogram of oriented gradient;

obtain a correspondence cost of the H-LF images based on the spectral-invariable feature descriptor;

perform H-LF stereo matching on the H-LF images to obtain a disparity map of a reference view; and

generate hyperspectral data-cubes by using the disparity map of the reference view.

22. The system of claim 21, wherein the data processing unit is configured to compare the spectral-invariant feature descriptor to develop a similarity metric using a bidirectional weighted normalized cross correlation (BWNCC).

23. The system of claim 21, wherein the data processing unit is configured to:

obtain an estimated spectra by synthesizing RGB color values from the H-LF images and mapping the RGB color values to a spectral band; and

obtain a defocus cost of the H-LF images based on a comparison between estimated spectra and captured spectra of the H-LF images.