Capturing and processing of images captured by camera arrays incorporating cameras with telephoto and conventional lenses to generate depth maps

• US 10,306,120 B2
• Filed: 03/13/2017
• Issued: 05/28/2019
• Est. Priority Date: 11/20/2009
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Duparre, Jacques
• Primary Examiner(s): Pinkney, Dawayne
• Application Number:
  US15/456,931
• Publication Number:
  US 20170187933A1
• Time in Patent Office: 806 Days
• Field of Search: 359618-621, 359625-626, 359443, 359454-455, 264 11, 264 132, 264 27, 3482221, 3482291, 3482312, 34823199, 348234, 348263, 348290, 348335, 348340, 348362
• CPC Class Codes: G02B 13/001 Miniaturised objectives for electronic devices, e.g. portable telephones, webcams, PDAs, small digital cameras G02B23/243 takes precedence
• CPC Class Codes: G02B 3/0056 arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085
• CPC Class Codes: G02B 3/0062 Stacked lens arrays, i.e. refractive surfaces arranged in at least two planes, without structurally separate optical elements in-between
• CPC Class Codes: G02B 5/201 in the form of arrays
• CPC Class Codes: H01L 27/14621 Colour filter arrangements
• CPC Class Codes: H01L 27/14625 Optical elements or arrangements associated with the device
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/16 Optical arrangements associated therewith, e.g. for beam-splitting or for colour correction
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/41 Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors
• CPC Class Codes: H04N 25/76 Addressed sensors, e.g. MOS or CMOS sensors

Abstract

Systems and methods for implementing array cameras configured to perform super-resolution processing to generate higher resolution super-resolved images using a plurality of captured images and lens stack arrays that can be utilized in array cameras are disclosed. Lens stack arrays in accordance with many embodiments of the invention include lens elements formed on substrates separated by spacers, where the lens elements, substrates and spacers are configured to form a plurality of optical channels, at least one aperture located within each optical channel, at least one spectral filter located within each optical channel, where each spectral filter is configured to pass a specific spectral band of light, and light blocking materials located within the lens stack array to optically isolate the optical channels.

Claims

1. A camera array, comprising:

a plurality of cameras including at least one camera comprising a telephoto lens and at least one camera comprising a wider field of view lens, where the plurality of cameras are configured to capture a plurality of images comprising different images representing different viewpoints of a same scene, wherein the camera array comprises a one-dimensional array of cameras;

control circuitry for independently controlling imaging parameters and operation of each camera in the plurality of cameras, wherein at least two cameras in the plurality of cameras have similar integration time settings; and

a parallax confirmation and measurement module for detecting and metering parallax using pixel correlation across cameras with similar integration time conditions; and

an image processing pipeline module configured to generate a depth map using the parallax information.

2. The camera array of claim 1, wherein the control circuitry provides a master setting for the imaging parameters and a deviation from the master setting is configured for each camera in the plurality of cameras.

3. The camera array of claim 2, wherein the master setting is selected from the group consisting of a gain setting, an integration time setting, and a digital processing setting.

4. The camera array of claim 1, wherein the at least one camera comprising the telephoto lens has other imaging characteristics that are the same as the at least one camera with the wider field of view lens.

5. The camera array of claim 1, wherein the at least one camera comprising the telephoto lens has different imaging parameters from the at least one camera comprising the wider field of view lens.

6. The camera array of claim 1, wherein the image processing pipeline module is further configured to generate a super-resolution telephoto image using the plurality of images captured by the plurality of cameras.

7. The camera array of claim 6, wherein the image processing pipeline module is configured to generate the at least super-resolution telephoto image by fusing the aligned image portions.

8. The camera array of claim 7, wherein the image pipeline processing module is further configured to interpolate luminance information from the aligned image portions to each grid point on the super-resolution telephoto image.

9. The camera array of claim 8, wherein the image processing pipeline module is further configured to perform super-resolution processing on the fused image portions to synthesize the super-resolution telephoto image.

10. The camera array of claim 6, further comprising:

an address conversion module configured to normalize the plurality of images captured by the plurality of cameras; and

an image pixel correlation module configured to align portion of images captured by different cameras to compensate for parallax based upon the detected and metered parallax information;

wherein the image processing pipeline module is configured to generate a higher resolution super-resolved image having a resolution that is higher than the resolutions of the plurality of images captured by the plurality of cameras.

11. The camera array of claim 1, wherein the parallax confirmation and measurement module is configured to detect and meter parallax by determining the parallax that yields the highest correlation between pixels from images captured by the plurality of cameras accounting for the positions of the cameras that captured the images.

12. The camera array of claim 1, wherein the parallax confirmation and measurement module is configured to determine the parallax that yields the highest correlation between pixels from images captured by performing pair-wise measurements to determine pixel correlation for different parallax-induced shifts.

13. The camera array of claim 12, wherein the parallax confirmation and measurement module is configured to determining the parallax that yields the highest correlations between pixels from images captured by keeping track of various pair-wise measurements and calculating a parallax that yields the highest correlation as the best least squares fit of the pair-wise measurements.

14. The camera array of claim 1, wherein the camera array comprises two or more cameras equipped with lenses of more than two magnifications to provide differing zoom magnifications.

15. The camera array of claim 1, wherein the camera array is an n×

n (n>

2) array camera.

16. The camera array of claim 1, wherein the control circuitry triggers each camera independently.

17. The camera array of claim 1, wherein the control circuitry triggers each camera in a synchronized manner.

18. The camera array of claim 1, wherein sensor elements in each of at least two cameras have different sizes.

19. The camera array of claim 1, wherein imaging parameters controlled by the control circuitry include exposure time, gain, and black level offset.