MULTISPECTRAL IMAGING SYSTEM

• US 20130250123A1
• Filed: 10/30/2012
• Published: 09/26/2013
• Est. Priority Date: 11/04/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Zhang, Xiaopeng, Aleksic, Milivoje, Vaddadi, Sundeep, Qiu, Gang, Wang, Jilai, Zhou, Xiaoming, Shen, Liang, Zhou, Shaojie
• Granted Patent: US 9,692,991 B2
• Time in Patent Office: Days
• US Class Current: 348/164
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths

Abstract

Systems and methods for multispectral imaging are disclosed. The multispectral imaging system can include a near infrared (NIR) imaging sensor and a visible imaging sensor. The disclosed systems and methods can be implemented to improve alignment between the NIR and visible images. Once the NIR and visible images are aligned, various types of multispectral processing techniques can be performed on the aligned images.

Claims

1. An electronic method for aligning a near infrared (NIR) image with a visible image in a system including a NIR imaging sensor and a visible imaging sensor, the method comprising:

receiving a NIR image from the NIR imaging sensor;

receiving a visible image from the visible imaging sensor;

matching pixels in the NIR image with pixels in the visible image to form a plurality of matched pixel pairs;

calculating pixel disparities for each matched pixel pair;

assigning weights to each matched pixel pair based at least in part on the calculated pixel disparities for each matched pixel pair; and

aligning the NIR image with the visible image based at least in part on the assigned weights.

2. The method of claim 1, wherein matching pixels in the NIR image comprises calculating image descriptors for each pixel in the NIR image and the visible image based at least in part on image gradients in the NIR image and the visible image.

3. The method of claim 1, wherein calculating the pixel disparities for each matched pixel pair includes calculating a pixel separation distance in the visible and NIR images between the pixels in that matched pixel pair.

4. The method of claim 1, wherein assigning weights to each pair of matched pixels comprises segmenting the matched pixel pairs into a plurality of blocks and calculating a disparity-similarity value and a color-similarity value for each matched pixel pair.

5. The method of claim 1, wherein aligning the NIR image with the visible image is based at least in part on a difference of intensities between the NIR image and the visible image.

6. The method of claim 5, further comprising warping one of the NIR image and the visible image to a perspective of the other of the NIR image and the visible image based at least in part on the plurality of matched pixel pairs.

7. The method of claim 1, wherein aligning the NIR image with the visible image is based at least in part on the assigned weights that satisfy a homographic constraint of the NIR and visible images.

8. A multispectral imaging system comprising:

a near infrared (NIR) imaging sensor configured to capture a NIR image;

a visible light imaging sensor configured to capture a visible light image;

a dense matching module programmed to match pixels in the NIR image with pixels in the visible light image to form a plurality of matched pixel pairs;

a sparse matching module programmed to;

calculate pixel disparities for each matched pixel pair; and

assign weights to each matched pixel pair based at least in part on the calculated pixel disparities for each matched pixel pair;

an aligned image formation module programmed to align the NIR image with the visible image based at least in part on the assigned weights; and

a storage module programmed to store the aligned image on the system.

9. The system of claim 8, wherein the dense matching module is further programmed to calculate image descriptors for each pixel in the NIR image and the visible image based at least in part on image gradients in the NIR image and the visible image.

10. The system of claim 9, wherein the dense matching module is configured to match image descriptors in the NIR image with image descriptors in the visible image to form the plurality of matched pixel pairs.

11. The system of claim 8, wherein the sparse matching module is programmed to segment the matched pixel pairs into a plurality of blocks and calculate a disparity-similarity value and a color-similarity value for each matched pixel pair.

12. The system of claim 8, wherein the aligned image formation module is programmed to align the NIR image with the visible image based at least in part on a difference of intensities between the NIR image and the visible image.

13. The system of claim 8, wherein the aligned image formation module is programmed to align the NIR image with the visible image based at least in part on the assigned weights that satisfy a homographic constraint of the NIR and visible images.

14. The system of claim 8, further comprising a stereoscopic imaging module programmed to render a three-dimensional image based at least in part on a depth map computed from the NIR image and the visible image.

15. The system of claim 8, further comprising a hybrid imaging module programmed to process still image data from the visible imaging sensor at a first resolution and video image data from the NIR imaging sensor at a second resolution, the second resolution less than the first resolution.

16. The system of claim 8, wherein the system is a cellular telephone.

17. The system of claim 8, further comprising a multispectral information processing module programmed to process NIR image data captured from the NIR imaging sensor to enhance contrast in visible image data captured from the visible imaging sensor.

18. An imaging system, comprising:

means for receiving a NIR image from a NIR imaging sensor;

means for receiving a visible image from a visible imaging sensor;

means for matching pixels in the NIR image with pixels in the visible image to form a plurality of matched pixel pairs;

means for calculating pixel disparities for each matched pixel pair;

means for assigning weights to each matched pixel pair based at least in part on the calculated pixel disparities for each matched pixel pair; and

means for aligning the NIR image with the visible image based at least in part on the assigned weights.

19. The imaging system of claim 18, wherein the means for receiving the NIR image and the means for receiving the visible image include a communications module.

20. The imaging system of claim 18, wherein the pixel matching means includes a dense matching module.

21. The imaging system of claim 18, wherein the pixel disparities calculating means includes a sparse matching module.

22. The imaging system of claim 21, wherein the sparse matching module further includes the weight assigning means.

23. The imaging system of claim 18, wherein the aligning means includes an aligned image formation module.

24. A non-transitory computer-readable medium having stored thereon code that when executed performs a method comprising:

receiving a NIR image from the NIR imaging sensor;

receiving a visible image from the visible imaging sensor;

matching pixels in the NIR image with pixels in the visible image to form a plurality of matched pixel pairs;

calculating pixel disparities for each matched pixel pair;

assigning weights to each matched pixel pair based at least in part on the calculated pixel disparities for each matched pixel pair; and

aligning the NIR image with the visible image based at least in part on the assigned weights.