Camera array systems including at least one bayer type camera and associated methods

• US 9,924,142 B2
• Filed: 11/21/2013
• Issued: 03/20/2018
• Est. Priority Date: 11/21/2012
• Status: Active Grant

• Current Assignee: Omnivision Technologies Incorporated (Will Semiconductor Co., Ltd. Shanghai)
• Original Assignee: Omnivision Technologies Incorporated (Will Semiconductor Co., Ltd. Shanghai)
• Inventors: Ni, Yongshen, Liu, Guansong
• Primary Examiner(s): Beasley, Deirdre
• Application Number:
  US14/086,731
• Publication Number:
  US 20140139642A1
• Time in Patent Office: 1,580 Days
• Field of Search: 3482231, 348 48, 382154, 382162
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

A method for processing a plurality of images of a scene recorded from different vantage points, where the plurality of images includes a color reference image captured by a Bayer type camera and at least one additional image, the method including (a) registering at least a portion of the plurality of images, and (b) generating a unitary color image from the plurality of images, wherein color information of the unitary color image is determined exclusively from the color reference image.

Claims

1. A method for processing a plurality of images of a scene recorded from different vantage points, the plurality of images comprising a single color reference image captured by a single one and only one Bayer type camera and at least two gray scale images respectively captured by at least two monochrome cameras, the method comprising:

registering at least a portion of the plurality of images;

generating a unitary color image from the plurality of images, color information of the unitary color image being determined exclusively from the single color reference image captured by the one and only one Bayer type camera, said generating including

(a) setting intensity of each pixel of the unitary color image to a weighted average of corresponding pixels of each of the at least two gray scale images, and

(b) setting color of each pixel of the unitary color image to color of corresponding pixel of the single color reference image; and

generating a depth map based upon (a) the color reference image and the at least two gray scale images and (b) stereo disparity between the color reference image and the at least two gray scale images caused by the different vantage points.

2. The method of claim 1, wherein the unitary color image is registered to the color reference image.

3. The method of claim 2, further comprising registering the depth map to the unitary color image.

4. The method of claim 1, the step of registering comprising, for each pixel in the color reference image:

finding one or more corresponding locations in the at least two gray scale images, each corresponding location belonging to a different one of the at least two gray scale images and representing the same portion of the scene as the pixel in the color reference image; and

determining a single stereo disparity, caused by the different vantage points and associated with the pixel in the color reference image, based on all of the one or more corresponding locations.

5. The method of claim 4, the step of determining a single stereo disparity comprising, when a pixel in the color reference image has a plurality of corresponding locations:

calculating a plurality of stereo disparities, each being the shift of a respective one of the plurality of corresponding locations relative to the pixel in the color reference image;

invoking one or more known relationships between the plurality of stereo disparities to eliminate all but the single stereo disparity;

solving a system of equations, expressed by the single stereo disparity, known coordinates of the pixel in the color reference image, and known coordinates of all of the corresponding locations, to determine the single stereo disparity.

6. The method of claim 5, further comprising, deducing, for each pixel in the color reference image, a depth of a portion of the scene associated with the pixel in the color reference image from the single stereo disparity.

7. The method of claim 1, in the step of generating a unitary color image, gray scale information of the unitary color image being determined from the at least two gray scale images.

8. The method of claim 7, the at least two gray scale images and the unitary color image having higher spatial resolution than the color reference image.

9. The method of claim 1, the at least two gray scale images comprising a plurality of spectrally selective images, wherein (a) the plurality of spectrally selective images represents a respective plurality of portions of the spectral range covered by the color reference image, and (b) the plurality of portions, taken in combination, comprising the spectral range covered by the unitary color image.

10. The method of claim 9, the gray scale information of the unitary color image being determined from the plurality of spectrally selective images.

11. The method of claim 10, the plurality of spectrally selective images and the unitary color image having higher spatial resolution than the color reference image.

19. The method of claim 1, the at least two gray scale images including a first image captured by a first camera sensitive to red light only, a second image captured by a second camera sensitive to green light only, and a third image captured by a third camera sensitive to blue light only, the step of setting intensity of each pixel comprising setting intensity of each pixel of the unitary color image to a weighted average of (a) a corresponding pixel of the first image, (b) a corresponding pixel of the second image, and (c) a corresponding pixel of the third image.

20. The method of claim 1, the step of setting intensity comprising defining intensities of the unitary color image at pixel resolution greater than pixel resolution of the color reference image.

12. A camera system comprising:

a camera array comprising a single Bayer type camera and at least two monochrome cameras, the Bayer type camera and each of the at least two monochrome cameras being configured to capture a respective plurality of images of a scene from a respective plurality of different vantage points;

a processor;

a memory communicatively coupled to the processor;

an image combination module, comprising machine-readable instructions stored in the memory, that, when executed by the processor, generate a unitary color image of the scene with color information obtained exclusively from a single image captured by the single Bayer type camera, the image combination module being adapted to (a) obtain gray scale information of the unitary color image from images captured by the at least two monochrome cameras and (b) set intensity of each pixel of the unitary color image to a weighted average of intensities of corresponding pixels of images captured by each of the at least two monochrome cameras; and

a depth module comprising machine-readable instructions stored in the memory, that, when executed by the processor and based upon stereo disparity between the plurality of images caused by the respective plurality of different vantage points, generate a depth map of the scene from information obtained from both the Bayer type camera and the at least two monochrome cameras.

13. The system of claim 12, the at least two monochrome cameras comprising a plurality of spectrally selective cameras sensitive to a respective portion of the spectral range of the Bayer type camera, and the plurality of spectrally selective cameras, taken in combination, being sensitive to at least the spectral range of the Bayer type camera.

14. The system of claim 12, the depth module being configured to register the depth map to the unitary color image.

15. The system of claim 12, further comprising:

a registration module comprising machine-readable instructions stored in the memory, that, when executed by the processor, generate registration data, expressed in terms of a single stereo disparity, caused by the different vantage points, for each pixel in the image captured by the Bayer type camera, using calibration data stored in the memory, the calibration data comprising relations between stereo disparities associated with different pairs of cameras of the camera array.

21. The system of claim 12, the at least two monochrome cameras including a first camera sensitive to red light only, a second camera sensitive to green light only, and a third camera sensitive to blue light only.

16. A system for processing a plurality of images of a scene recorded from a respective plurality of different vantage points, the plurality of images comprising a single color reference image captured by one and only one Bayer type camera and at least two gray scale images captured by at least two respective monochrome cameras, the system comprising:

a processor;

a memory communicatively coupled to the processor;

an image combination module comprising machine-readable instructions stored in the memory, that, when executed by the processor, process the plurality of images to generate a unitary color image of the scene with color information obtained exclusively from the single color reference image captured by the one and only one Bayer type camera, the unitary color image being registered to the color reference image, the image combination module being adapted to set intensity of each pixel of the unitary color image to a weighted average of intensities of corresponding pixels of images captured by each of the at least two gray scale images; and

a depth module comprising machine-readable instructions stored in the memory, that, when executed by the processor and based upon stereo disparity between the plurality of images caused by the respective plurality of different vantage points, generate a depth map of the scene from the color reference image and the at least two gray scale images.

17. The system of claim 16,

the depth module further being configured to register the depth map to the color reference image.

18. The system of claim 16, further comprising:

a registration module comprising machine-readable instructions stored in the memory, that, when executed by the processor, generate registration data, expressed in terms of a single stereo disparity, caused by the different vantage points, for each pixel in the color reference image, using calibration data stored in the memory, the calibration data comprising relations between stereo disparities associated with different pairs of images from the plurality of images.

22. The system of claim 16, the at least two gray scale images including a first image captured by a first camera sensitive to red light only, a second image captured by a second camera sensitive to green light only, and a third image captured by a third camera sensitive to blue light only, the image combination module being adapted to set the intensity of each pixel of the unitary color image to a weighted average of (a) a corresponding pixel of the first image, (b) a corresponding pixel of the second image, and (c) a corresponding pixel of the third image.