Array camera configurations incorporating constituent array cameras and constituent cameras

• US 10,708,492 B2
• Filed: 11/08/2018
• Issued: 07/07/2020
• Est. Priority Date: 11/26/2013
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Venkataraman, Kartik, Duparre, Jacques
• Primary Examiner(s): Pasiewicz, Daniel M
• Application Number:
  US16/184,840
• Publication Number:
  US 20190098209A1
• Time in Patent Office: 607 Days
• CPC Class Codes: G02B 13/02 Telephoto objectives, i.e. systems of the type + - in which the distance from the front vertex to the image plane is less than the equivalent focal length
• CPC Class Codes: G02B 3/0056 arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085
• CPC Class Codes: G02B 3/0062 Stacked lens arrays, i.e. refractive surfaces arranged in at least two planes, without structurally separate optical elements in-between
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/702 SSIS architectures characterised by non-identical, non-equidistant or non-planar pixel layout
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Systems and methods for implementing array camera configurations that include a plurality of constituent array cameras, where each constituent array camera provides a distinct field of view and/or a distinct viewing direction, are described. In several embodiments, image data captured by the constituent array cameras is used to synthesize multiple images that are subsequently blended. In a number of embodiments, the blended images include a foveated region. In certain embodiments, the blended images possess a wider field of view than the fields of view of the multiple images.

Claims

1. An array camera configuration, comprising:

a camera array comprising a plurality of cameras each having a field of view, where each camera comprises optics that form an image on a focal plane defined by an array of pixels that capture image data, wherein a first subset of the plurality of cameras forms a first constituent array camera, the cameras in the first subset have a first combined field of view, and an object of interest is visible within the first combined field of view;

a processor; and

a memory containing an image processing application, wherein the image processing application directs the processor to;

obtain a first set of image data from the first constituent array camera;

identify the object of interest within the first set of image data;

select a second subset of the plurality of cameras to form a second constituent array camera, by maximizing the baseline between at least one camera in the first constituent array camera that captures at least a portion of the first set of image data and at least one camera in the second constituent array camera that captures at least a portion of a second set of image data, wherein the cameras in the second subset have a second combined field of view and the object of interest is visible within the second combined field of view;

obtain the second set of image data from the second constituent array camera;

identify the object of interest within the second set of image data; and

calculate depth information for the object of interest using the at least a portion of the first set of image data and the at least a portion of the second set of image data.

2. The array camera configuration of claim 1, wherein the image processing application further directs the processor to:

calculate preliminary depth information for the object of interest based on the first set of image data; and

select the second subset of the plurality of cameras to form a second constituent array camera when the preliminary depth information indicates that the object of interest is located within a predetermined distance from the first constituent array camera.

3. The array camera configuration of claim 1, wherein the image processing application directs the processor to calculate depth information by:

determining a disparity between a pixel associated with the object of interest in the first set of image data and a corresponding pixel in the second set of image data.

4. The array camera configuration of claim 1, wherein the image processing application directs the processor to calculate depth information for the object of interest further based on the distance between cameras in the first and second constituent array cameras that capture images used to calculate the depth information.

5. The array camera configuration of claim 1, wherein the image processing application further directs the processor to:

generate a depth map using the depth information, and the first and second sets of image data.

6. The array camera configuration of claim 1, wherein the image processing application further directs the processor to:

synthesize an image using the first and second sets of image data and the depth information.

7. The array camera configuration of claim 1, wherein the first set of image data includes image data with respect to a first portion of a scene, and the second set of image data includes image data with respect to a second portion of the scene, such that the first and second portions of the scene encompass the scene in its entirety.

8. The array camera configuration of claim 1, wherein the image processing application directs the processor to independently control each of the first and second constituent array cameras.

9. The array camera configuration of claim 1, wherein cameras in the first and second arrays of cameras have different imaging characteristics.

10. The array camera configuration of claim 1, wherein at least one of the first or second array of cameras includes a M×

N array of cameras.