Multispectral imaging system

• US 9,692,991 B2
• Filed: 10/30/2012
• Issued: 06/27/2017
• Est. Priority Date: 11/04/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Qiu, Gang, Vaddadi, Sundeep, Zhang, Xiaopeng, Shen, Liang, Aleksic, Milivoje, Wang, Jilai, Zhou, Xiaoming, Zhuo, Shaojie
• Primary Examiner(s): Geroleo, Francis G
• Application Number:
  US13/663,897
• Publication Number:
  US 20130250123A1
• Time in Patent Office: 1,701 Days
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths

Abstract

Systems and methods for multispectral imaging are disclosed. The multispectral imaging system can include a near infrared (NIR) imaging sensor and a visible imaging sensor. The disclosed systems and methods can be implemented to improve alignment between the NIR and visible images. Once the NIR and visible images are aligned, various types of multispectral processing techniques can be performed on the aligned images.

Claims

1. An electronic method for aligning a near infrared (NIR) image with a visible image in a system including a NIR imaging sensor and a visible imaging sensor, the method comprising:

receiving a NIR image from the NIR imaging sensor, the NIR image being devoid of visible image data;

receiving a visible image from the visible imaging sensor;

matching pixels in the NIR image with pixels in the visible image on a pixel-by-pixel basis to form a plurality of matched pixel pairs based at least in part on an association of NIR pixel image data with visible pixel image data for each matched pixel pair;

calculating a pixel disparity, a disparity-similarity, and a color-similarity for each matched pixel pair;

assigning weights to each matched pixel pair based at least in part on the calculated pixel disparity, the calculated disparity-similarity, and the calculated color-similarity for each matched pixel pair; and

aligning the NIR image with the visible image based at least in part on the assigned weights that exceed a predetermined threshold.

2. The electronic method of claim 1, wherein matching pixels in the NIR image comprises calculating image descriptors for each pixel in the NIR image and the visible image based at least in part on image gradients in the NIR image and the visible image.

3. The electronic method of claim 1, wherein calculating the pixel disparities for each matched pixel pair includes calculating a pixel separation distance in the visible and NIR images between the pixels in that matched pixel pair.

4. The electronic method of claim 1, wherein assigning weights to each pair of matched pixels comprises segmenting the matched pixel pairs into a plurality of blocks.

5. The electronic method of claim 1, wherein aligning the NIR image with the visible image is based at least in part on a difference of intensities between the NIR image and the visible image.

6. The electronic method of claim 5, further comprising warping one of the NIR image and the visible image to a perspective of the other of the NIR image and the visible image based at least in part on the plurality of matched pixel pairs.

7. The electronic method of claim 1, wherein aligning the NIR image with the visible image is based at least in part on the assigned weights that satisfy a homographic constraint of the NIR and visible images.

8. A multispectral imaging system comprising:

a near infrared (NIR) imaging sensor configured to capture a NIR image, the NIR image being devoid of visible image data;

a visible light imaging sensor configured to capture a visible light image;

a dense matching module programmed to match pixels in the NIR image with pixels in the visible light image on a pixel-by-pixel basis to form a plurality of matched pixel pairs based at least in part on an association of NIR pixel image data with visible pixel image data for each matched pixel pair;

a sparse matching module programmed to;

calculate a pixel disparity, a disparity-similarity, and a color-similarity for each matched pixel pair; and

assign weights to each matched pixel pair based at least in part on the calculated pixel disparity, the calculated disparity-similarity, and the calculated color-similarity for each matched pixel pair;

an aligned image formation module programmed to align the NIR image with the visible image based at least in part on the assigned weights that exceed a predetermined threshold; and

a storage module programmed to store the aligned image on the system.

9. The multispectral imaging system of claim 8, wherein the dense matching module is further programmed to calculate image descriptors for each pixel in the NIR image and the visible image based at least in part on image gradients in the NIR image and the visible image.

10. The multispectral imaging system of claim 9, wherein the dense matching module is configured to match image descriptors in the NIR image with image descriptors in the visible image to form the plurality of matched pixel pairs.

11. The multispectral imaging system of claim 8, wherein the sparse matching module is programmed to segment the matched pixel pairs into a plurality of blocks.

12. The multispectral imaging system of claim 8, wherein the aligned image formation module is programmed to align the NIR image with the visible image based at least in part on a difference of intensities between the NIR image and the visible image.

13. The multispectral imaging system of claim 8, wherein the aligned image formation module is programmed to align the NIR image with the visible image based at least in part on the assigned weights that satisfy a homographic constraint of the NIR and visible images.

14. The multispectral imaging system of claim 8, further comprising a stereoscopic imaging module programmed to render a three-dimensional image based at least in part on a depth map computed from the NIR image and the visible image.

15. The multispectral imaging system of claim 8, further comprising a hybrid imaging module programmed to process still image data from the visible imaging sensor at a first resolution and video image data from the NIR imaging sensor at a second resolution, the second resolution less than the first resolution.

16. The multispectral imaging system of claim 8, wherein the multispectral imaging system is a cellular telephone.

17. The multispectral imaging system of claim 8, further comprising a multispectral information processing module programmed to process NIR image data captured from the NIR imaging sensor to enhance contrast in visible image data captured from the visible imaging sensor.

18. An imaging system, comprising:

means for receiving a near infrared (NIR) image from a NIR imaging sensor, the NIR image being devoid of visible image data;

means for receiving a visible image from a visible imaging sensor;

means for matching pixels in the NIR image with pixels in the visible image on a pixel-by-pixel basis to form a plurality of matched pixel pairs based at least in part on an association of NIR pixel image data with visible pixel image data for each matched pixel pair;

means for calculating a pixel disparity, a disparity-similarity, and a color-similarity for each matched pixel pair;

means for assigning weights to each matched pixel pair based at least in part on the calculated pixel disparity, the calculated disparity-similarity, and the calculated color-similarity for each matched pixel pair; and

means for aligning the NIR image with the visible image based at least in part on the assigned weights that exceed a predetermined threshold.

19. The imaging system of claim 18, wherein the means for receiving the NIR image and the means for receiving the visible image include a communications module.

20. The imaging system of claim 18, wherein the pixel matching means includes a dense matching module.

21. The imaging system of claim 18, wherein the pixel disparities calculating means includes a sparse matching module.

22. The imaging system of claim 21, wherein the sparse matching module further includes the weight assigning means.

23. The imaging system of claim 18, wherein the aligning means includes an aligned image formation module.

24. A non-transitory computer-readable medium having stored thereon code that when executed performs a method comprising:

receiving a near infrared (NIR) image from the NIR imaging sensor, the NIR image being devoid of visible image data;

receiving a visible image from the visible imaging sensor;

matching pixels in the NIR image with pixels in the visible image on a pixel-by-pixel basis to form a plurality of matched pixel pairs based at least in part on an association of NIR pixel image data with visible pixel image data for each matched pixel pair;

calculating a pixel disparity, a disparity-similarity, and a color-similarity for each matched pixel pair;

assigning weights to each matched pixel pair based at least in part on the calculated pixel disparity, the calculated disparity-similarity, and the calculated color-similarity for each matched pixel pair; and

aligning the NIR image with the visible image based at least in part on the assigned weights that exceed a predetermined threshold.