Array camera configurations incorporating constituent array cameras and constituent cameras

• US 9,813,617 B2
• Filed: 09/23/2016
• Issued: 11/07/2017
• Est. Priority Date: 11/26/2013
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Fotonation Cayman Limited
• Inventors: Venkataraman, Kartik, Duparr, Jacques
• Primary Examiner(s): Pasiewicz, Daniel M
• Application Number:
  US15/274,830
• Publication Number:
  US 20170078568A1
• Time in Patent Office: 410 Days
• CPC Class Codes: G02B 13/02 Telephoto objectives, i.e. systems of the type + - in which the distance from the front vertex to the image plane is less than the equivalent focal length
• CPC Class Codes: G02B 3/0056 arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085
• CPC Class Codes: G02B 3/0062 Stacked lens arrays, i.e. refractive surfaces arranged in at least two planes, without structurally separate optical elements in-between
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/702 SSIS architectures characterised by non-identical, non-equidistant or non-planar pixel layout
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Systems and methods for implementing array camera configurations that include a plurality of constituent array cameras, where each constituent array camera provides a distinct field of view and/or a distinct viewing direction, are described. In several embodiments, image data captured by the constituent array cameras is used to synthesize multiple images that are subsequently blended. In a number of embodiments, the blended images include a foveated region. In certain embodiments, the blended images possess a wider field of view than the fields of view of the multiple images.

Claims

1. An array camera configuration, comprising:

a first array of cameras comprising a first set of cameras each having a field of view, wherein the cameras in the first array of cameras have a first combined field of view that encompasses at least a portion of the field of view of each of the cameras in the first array of cameras;

a second array of cameras comprising a second set of cameras each having a field of view, wherein the cameras in the second array of cameras have a second combined field of view that encompasses at least a portion of the field of view of each of the cameras in the second array of cameras, the majority of the second combined field of view being adjacent to the first combined field of view;

a processor;

a memory containing an image processing application, wherein the image processing application directs the processor to;

obtain a first set of image data from the first array of cameras and a second set of image data from the second array of cameras;

synthesize a first image using the first set of image data and a second image using the second set of image data, the second image having a viewpoint; and

warp the first image into the viewpoint of the second image to form a warped first image; and

combine the warped first image with the second image to form a combined image with a field of view encompassing at least a portion of the first combined field of view and at least a portion of the second combined field of view.

2. The array camera configuration of claim 1, wherein the image processing application further directs the processor to:

generate a depth map using the first set of image data; and

wherein warping of the first image into the viewpoint of the second image is performed using the depth map.

3. The array camera configuration of claim 1, wherein combining the warped first image and the second image is performed using interpolation.

4. The array camera configuration of claim 1, wherein the first combined field of view overlaps with a portion of the second combined field of view.

5. The array camera configuration of claim 1, wherein the first set of image data includes image data with respect to a first portion of a scene, and the second set of image data includes image data with respect to a second portion of the scene, such that the first and second portions of the scene encompass the scene in its entirety.

6. The array camera configuration of claim 1, further comprising:

a third array of cameras comprising a third set of cameras each having a field of view, wherein the cameras in the third array of cameras have a third combined field of view that encompasses at least a portion of the field of view of each of the cameras in the third array of cameras, the majority of the third combined field of view being adjacent to the second combined field of view;

wherein the second field of view images a central portion of a scene, and the first and third arrays of cameras are disposed on either side of the second array of cameras.

7. The array camera configuration of claim 1, wherein the image processing application directs the processor to independently control each of the first and second arrays of cameras.

8. The array camera configuration of claim 7, wherein the image processing application directs the processor to control the first and second arrays of cameras to utilize different exposures.

9. The array camera configuration of claim 7, wherein the image processing application directs the processor to control at least one of the first or second array of cameras to utilize binning.

10. The array camera configuration of claim 1, wherein the first combined field of view includes a region that is common to the fields of view of multiple cameras in the first array of cameras, and the second combined field of view includes a region that is common to the fields of view of multiple cameras in the second array of cameras.

11. The array camera configuration of claim 1, wherein the image processing application further directs the processor to:

generate a first depth map using the first set of image data; and

generate a second depth map using the second set of image data.

12. The array camera configuration of claim 11, wherein the image processing application further directs the processor to:

combine the first and second depth maps to create a combined depth map for the combined image.

13. The array camera configuration of claim 1, wherein the image processing application further directs the processor to:

generate a depth map using the first and second sets of image data.

14. The array camera configuration of claim 1, wherein cameras in the first and second arrays of cameras have different imaging characteristics.

15. The array camera configuration of claim 1, wherein at least one of the first or second array of cameras includes a M×

N array of cameras.