Camera Array Systems Including At Least One Bayer Type Camera And Associated Methods

• US 20140139642A1
• Filed: 11/21/2013
• Published: 05/22/2014
• Est. Priority Date: 11/21/2012
• Status: Active Grant

• Current Assignee: Omnivision Technologies Incorporated (Will Semiconductor Co., Ltd. Shanghai)
• Original Assignee: Omnivision Technologies Incorporated (Will Semiconductor Co., Ltd. Shanghai)
• Inventors: Liu, Guansong, Ni, Yongshen
• Granted Patent: US 9,924,142 B2
• Time in Patent Office: Days
• US Class Current: 348/48
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

A method for processing a plurality of images of a scene recorded from different vantage points, where the plurality of images includes a color reference image captured by a Bayer type camera and at least one additional image, the method including (a) registering at least a portion of the plurality of images, and (b) generating a unitary color image from the plurality of images, wherein color information of the unitary color image is determined exclusively from the color reference image.

Claims

1. A method for processing a plurality of images of a scene recorded from different vantage points, the plurality of images comprising a color reference image captured by a Bayer type camera and at least one additional image, the method comprising:

registering at least a portion of the plurality of images; and

generating a unitary color image from the plurality of images, color information of the unitary color image being determined exclusively from the color reference image.

2. The method of claim 1, wherein the unitary color image is registered to the color reference image.

3. The method of claim 2, further comprising:

generating a depth map registered to the unitary color image.

4. The method of claim 1, the step of registering comprising, for each pixel in the color reference image:

finding one or more corresponding locations in the at least one additional image, each corresponding location belonging to a different one of the at least one additional image and representing the same portion of the scene as the pixel in the color reference image; and

determining a single stereo disparity, associated with the pixel in the color reference image, based on all of the one or more corresponding locations.

5. The method of claim 4, the step of determining a single stereo disparity comprising, when a pixel in the color reference image has a plurality of corresponding locations:

calculating a plurality of stereo disparities, each being the shift of a respective one of the plurality of corresponding locations relative to the pixel in the color reference image;

invoking one or more known relationships between the plurality of stereo disparities to eliminate all but the single stereo disparity;

solving a system of equations, expressed by the single stereo disparity, known coordinates of the pixel in the color reference image, and known coordinates of all of the corresponding locations, to determine the single stereo disparity.

6. The method of claim 5, further comprising, deducing, for each pixel in the color reference image, a depth of a portion of the scene associated with the pixel in the color reference image from the single stereo disparity.

7. The method of claim 1, the at least one additional image comprising at least one gray scale image.

8. The method of claim 7, gray scale information of the unitary color image being determined from the at least one gray scale image.

9. The method of claim 8, the at least one gray scale image and the unitary color image having higher spatial resolution than the color reference image.

10. The method of claim 1, the at least one additional image comprising a plurality of spectrally selective images, wherein (a) the plurality of spectrally selective images represents a respective plurality of portions of the spectral range covered by the color reference image, and (b) the plurality of portions, taken in combination, comprising the spectral range covered by the color image.

11. The method of claim 10, the gray scale information of the unitary color image being determined from the plurality of spectrally selective images.

12. The method of claim 11, the plurality of spectrally selective images and the unitary color image having higher spatial resolution than the color reference image.

13. A camera system comprising:

a camera array for capturing a plurality of images of a scene from a plurality of vantage points, the camera array comprising at least one color camera with a Bayer type color filter array;

a processor;

a memory communicatively coupled to the processor; and

an image combination module, comprising machine-readable instructions stored in the memory, that, when executed by the processor, generate a unitary color image of the scene with color information obtained exclusively from the image captured by a selected one of the at least one color camera.

14. The system of claim 13,

wherein the camera array further comprises at least one monochrome camera;

and

wherein the image combination module is further adapted to obtain gray scale information of the unitary color image from images captured by the at least one monochrome camera.

15. The system of claim 13,

wherein the camera array further comprises a plurality of spectrally selective cameras sensitive to a respective portion of the spectral range of the selected one of the at least one color camera, and the plurality of spectrally selective cameras, taken in combination, is sensitive to at least the spectral range of the selected one of the at least one color camera;

and

wherein the image combination module is further adapted to obtain gray scale information of the unitary color image from images captured by the plurality of spectrally selective cameras.

16. The system of claim 13, further comprising:

a depth module, comprising machine-readable instructions stored in the memory, that, when executed by the processor, generate a depth map of the scene.

17. The system of claim 16, the depth map being registered to the unitary color image.

18. The system of claim 13, further comprising:

a registration module, comprising machine-readable instructions stored in the memory, that, when executed by the processor, generate registration data, expressed in terms of a single stereo disparity for each pixel in the image captured by the selected one of the at least one color camera, using calibration data stored in the memory, the calibration data comprising relations between stereo disparities associated with different pairs of cameras of the camera array.

19. A system for processing a plurality of images of a scene recorded from different vantage points, the plurality of images comprising a color reference image captured by a Bayer type camera and at least one additional image, the system comprising:

a processor;

a memory communicatively coupled to the processor; and

an image combination module, comprising machine-readable instructions stored in the memory, that, when executed by the processor, generate a unitary color image of the scene with color information obtained exclusively from the color reference image, the unitary color image being registered to the color reference image.

20. The system of claim 19, further comprising:

a depth module, comprising machine-readable instructions stored in the memory, that, when executed by the processor, generate a depth map of the scene, the depth map being registered to the color reference image.

21. The system of claim 19, further comprising:

a registration module, comprising machine-readable instructions stored in the memory, that, when executed by the processor, generate registration data, expressed in terms of a single stereo disparity for each pixel in the color reference image, using calibration data stored in the memory, the calibration data comprising relations between stereo disparities associated with different pairs of images from the plurality of images.