Extended color processing on pelican array cameras

• US 10,638,099 B2
• Filed: 01/11/2019
• Issued: 04/28/2020
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Mullis, Robert H., Lelescu, Dan, Venkataraman, Kartik
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): Walsh, Kathleen M
• Application Number:
  US16/246,100
• Publication Number:
  US 20190215496A1
• Time in Patent Office: 473 Days
• Field of Search: 348164
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 23/15 Image signal generation with circuitry for avoiding or correcting image misregistration
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Systems and methods for extended color processing on Pelican array cameras in accordance with embodiments of the invention are disclosed. In one embodiment, a method of generating a high resolution image includes obtaining input images, where a first set of images includes information in a first band of visible wavelengths and a second set of images includes information in a second band of visible wavelengths and non-visible wavelengths, determining an initial estimate by combining the first set of images into a first fused image, combining the second set of images into a second fused image, spatially registering the fused images, denoising the fused images using bilateral filters, normalizing the second fused image in the photometric reference space of the first fused image, combining the fused images, determining a high resolution image that when mapped through a forward imaging transformation matches the input images within at least one predetermined criterion.

Claims

1. A method of generating an image of a scene using a camera array including at least one camera that captures an RGB image of a scene and at least one camera that captures near-infrared (IR) spectral wavelengths of the scene, the method comprising:

obtaining input images captured by a plurality of cameras that includes a camera that captures an RGB image and a camera that captures near-IR wavelengths, where the input images includes a first input image that includes image information captured in at least three channels (RGB) of information and a second input image that includes image information captured in at least a near-IR channel of information;

generate a fused image using a processor configured by software to;

measure parallax using the input images captured by the plurality of cameras to produce a depth map;

normalize the second input image in the photometric reference space of the first input image;

cross-channel normalize the first input image with respect to the second input image by applying gains and offsets to pixels of the first input image; and

perform cross-channel fusion using the first input image and the second input image to produce an image.

2. The method of claim 1, wherein the first input image and the second input image have the same resolution.

3. The method of claim 1, further comprising:

capturing a first set of input images that are captured by a first set of RGB cameras from the plurality of cameras; and

capturing a second set of input images that are captured by a second near-IR camera from the plurality of cameras.

4. The method of claim 3, further comprising:

combining image information from the first set of input images into a first fused image using analog gain and noise information from the first set of RGB cameras; and

combining image information from the second set of input images into a second fused image utilizes using analog gain and noise information from the second near-IR camera.

5. The method of claim 1, further comprising denoising the first input image using a first bilateral filter and denoising the second input image using a second bilateral filter, wherein the first bilateral filter and the second bilateral filter utilize weights that are a function of both the photometric and geometric distance between a pixel and pixels in the neighborhood of the pixel.

6. The method of claim 5, wherein the first bilateral filter is a cross-channel bilateral filter utilizing weights determined for the second input image.

7. The method of claim 5, wherein the first input image is captured by a first camera from the plurality of cameras and the first bilateral filter is a cross-channel bilateral filter utilizing weights determined for the second input image when an analog gain value of the first camera is above a predetermined threshold.

8. The method of claim 1, wherein normalizing the second input image in the photometric reference space of the first input image comprises applying gains and offsets to pixels of the second input image.

9. The method of claim 1, wherein the processor being configured to normalize the second input image in the photometric reference space of the first input image comprises the processor being configured to:

select a first pixel of interest in the second input image and a first collection of similar pixels in the neighborhood of the first pixel of interest;

select a second pixel of interest in the first input image corresponding to the first pixel of interest and a second collection of similar pixels in the neighborhood of the second pixel of interest;

determine the intersection of the first collection of similar pixels and the second collection of similar pixels;

calculate gain and offset values using the intersection of the two collections;

apply the gain and offset values to the appropriate pixels in the second input image.

10. The method of claim 9, where the intersection of the first collection of similar pixels and the second collection of similar pixels is the set of pixels in the first and second collections having the same corresponding locations in each of the first input image and the second input image.

11. An array camera configured to generate an image of a scene using an array camera including at least one camera that captures an RGB image of a scene and at least one camera that captures at least near-infrared (IR) spectral wavelengths of the scene, the array camera comprising:

an array camera including a plurality of cameras that includes a camera that captures an RGB image and a camera that captures at least near-IR spectral wavelengths; and

a processor configured by software to;

obtain input images captured by the plurality of cameras that includes the camera that captures an RGB image and the camera that captures at least near-IR spectral wavelengths, where the input images includes a first input image that includes image information captured in at least three channels (RGB) of information and a second input image that includes image information captured in a near-IR channel of information;

generate a fused image by;

measuring parallax using the input images captured by the plurality of cameras to produce a depth map;

normalizing the second input image in the photometric reference space of the first input image;

cross-channel normalize the first input image with respect to the second input image by applying gains and offsets to pixels of the first input image; and

perform cross-channel fusion using the first input image the second input image to produce an image.

12. The array camera of claim 11, wherein the three channels (RGB) of information includes green and red light.

13. The array camera of claim 11, wherein the first input image and the second input image have the same resolution.

14. The array camera of claim 11, wherein the processor is further configured to:

capture a first set of input images that are captured by a first set of RGB cameras from the plurality of cameras; and

capture a second set of input images that are captured by a second near-IR camera from the plurality of cameras.

15. The array camera of claim 14, wherein the processor is further configured to:

combine image information from the first set of input images into a first fused image using analog gain and noise information from the first set of RGB cameras; and

combine image information from the second set of input images into a second fused image utilizes using analog gain and noise information from the second near-IR camera.

16. The array camera of claim 11, wherein the processor is further configured to denoise the first input image using a first bilateral filter and denoise the second input image using a second bilateral filter, wherein the first bilateral filter and the second bilateral filter utilize weights that are a function of both the photometric and geometric distance between a pixel and pixels in the neighborhood of the pixel.

17. The array camera of claim 16, wherein the first bilateral filter is a cross-channel bilateral filter utilizing weights determined for the second input image.

18. The array camera of claim 16, wherein the first input image is captured by a first camera from the plurality of cameras and the first bilateral filter is a cross-channel bilateral filter utilizing weights determined for the second input image when an analog gain value of the first camera is above a predetermined threshold.

19. The array camera of claim 11, wherein normalizing the second input image in the photometric reference space of the first input image comprises applying gains and offsets to pixels of the second input image.