Capturing and processing of images using monolithic camera array with heterogeneous imagers

• US 8,866,920 B2
• Filed: 11/22/2010
• Issued: 10/21/2014
• Est. Priority Date: 05/20/2008
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Pelican Imaging Corporation
• Inventors: Venkataraman, Kartik, Jabbi, Amandeep S., Mullis, Robert H., Hu, Shane Ching-Feng, Duparre, Jacques
• Primary Examiner(s): LAM, HUNG H
• Application Number:
  US12/952,134
• Publication Number:
  US 20110080487A1
• Time in Patent Office: 1,429 Days
• Field of Search: 348/208.1, 348/221.2, 348/2, 348/229.1, 348/234, 348/362
• US Class Current: 348/218.1
• CPC Class Codes: G02B 13/0015 characterised by the lens design
• CPC Class Codes: G02B 3/0056 arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085
• CPC Class Codes: G02B 3/0062 Stacked lens arrays, i.e. refractive surfaces arranged in at least two planes, without structurally separate optical elements in-between
• CPC Class Codes: G02B 5/20 Filters polarising elements G02B5/30
• CPC Class Codes: G02B 5/201 in the form of arrays
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 23/16 Optical arrangements associated therewith, e.g. for beam-splitting or for colour correction
• CPC Class Codes: H04N 23/45 : for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/54 : Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/55 : Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 23/69 : Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/88 : for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/951 : by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/131 : including elements passing infrared wavelengths
• CPC Class Codes: H04N 25/133 : including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 25/135 : based on four or more different wavelength filter elements
• CPC Class Codes: H04N 25/41 : Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors
• CPC Class Codes: H04N 25/48 : Increasing resolution by shifting the sensor relative to the scene
• CPC Class Codes: H04N 25/581 : acquired simultaneously
• CPC Class Codes: H04N 25/60 : Noise processing, e.g. detecting, correcting, reducing or removing noise
• CPC Class Codes: H04N 25/6153 : for colour signals
• CPC Class Codes: H04N 25/67 : applied to fixed-pattern noise, e.g. non-uniformity of response
• CPC Class Codes: H04N 25/705 : Pixels for depth measurement, e.g. RGBZ
• CPC Class Codes: H04N 25/79 : Arrangements of circuitry being divided between different or multiple substrates, chips or circuit boards, e.g. stacked image sensors
• CPC Class Codes: H04N 5/262 : Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 5/265 : Mixing

Abstract

Systems and methods for implementing array cameras configured to perform super-resolution processing to generate higher resolution super-resolved images using a plurality of captured images and lens stack arrays that can be utilized in array cameras are disclosed. An imaging device in accordance with one embodiment of the invention includes at least one imager array, and each imager in the array comprises a plurality of light sensing elements and a lens stack including at least one lens surface, where the lens stack is configured to form an image on the light sensing elements, control circuitry configured to capture images formed on the light sensing elements of each of the imagers, and a super-resolution processing module configured to generate at least one higher resolution super-resolved image using a plurality of the captured images.

Claims

1. An imaging device comprising:

at least one imager array comprising a plurality of imagers, and each imager in the array comprises a plurality of light sensing elements and a lens stack including at least one lens surface, where the lens stack is configured to form an image on the light sensing elements;

control circuitry configured to capture low resolution images formed on the light sensing elements of the imagers, where;

the low resolution images captured by different imagers include different occlusion sets;

the occlusion set of a specific low resolution image captured by a given imager is the portion of a scene visible to a baseline imager from plurality of imagers that is occluded from the view of the given imager; and

the plurality of imagers includes a first imager that captures pixels around an edge of a foreground object that is visible to the baseline imagers and is in the occlusion set of a second of the plurality of imagers that has the same imaging characteristics as the first imager; and

a super-resolution processing module configured to generate at least one higher resolution super-resolved image using a plurality of the captured low resolution images, where the super-resolution processing module is configured to perform parallax detection and generates parallax information to align portions of the plurality of captured low resolution images captured by different imagers and the parallax information includes a depth map.

2. The imaging device of claim 1, wherein:

a spectral filter is provided within each imager and is configured to pass a specific spectral band of light; and

spectral filters that pass different spectral bands are provided within at least two of the imagers.

3. The imaging device of claim 2, wherein the spectral filter is selected from the group consisting of an organic color filter, an absorptive material, a dielectric coating, an interference filter, a multilayer coating, and combinations thereof.

4. The imaging device of claim 2, wherein at least one of the imagers further comprises a polarizing filter.

5. The imaging device of claim 2, wherein the construction of the lens stack within an imager differs based upon the specific spectral band of light passed by the spectral filter within the imager such that chromatic aberrations are reduced.

6. The imaging device of claim 5, wherein the prescription of at least one surface of the lens stack within an imager is a function of the specific spectral band of light passed by the spectral filter within the imager.

7. The imaging device of claim 6, wherein the at least one surface of the lens stack is configured so that the back focal length of each lens stack in the imager array are the same independent of the spectral band.

8. The imaging device of claim 2, wherein a combination of high and low Abbe number materials is used in the construction of each of the lens stacks to reduce chromatic aberrations.

9. The imaging device of claim 2, wherein each lens stack includes at least one aperture stop positioned in front of the light sensing element, and the spectral filter is positioned in proximity to the aperture stop.

10. The imaging device of claim 9, wherein the at least one aperture stop is formed by a light blocking material selected from the group consisting of metal materials, oxide materials, black particle filled photoresists and combinations thereof.

11. The imaging device of claim 2, wherein:

at least one lens surface in the optical stack of an imager differs based upon the specific spectral band of light passed by the spectral filter within the imager; and

each lens surface is selected from the group consisting of diffractive, Fresnel, refractive and combinations thereof.

12. The imaging device of claim 11, wherein the radii of curvature of the lens surfaces differ based upon the specific spectral band of light passed by the spectral filter within the imager.

13. The imaging device of claim 11, wherein the lens stacks of each of the imagers have the same back focal length.

14. The imaging device of claim 1, wherein the light sensing elements are disposed to detect an image in a backside imaging mode.

15. The imaging device of claim 14, wherein a separate negative lens element is disposed in close proximity to the light sensing elements of an imager and in optical alignment with the lens stack of the imager so that the field curvature of the lens stack is corrected.

16. The imaging device of claim 1, wherein a separate negative lens element is disposed in close proximity to the light sensing elements of an imager and in optical alignment with the lens stack of the imager so that the field curvature of the lens stack is corrected.

17. The imaging device of claim 1, wherein at least two of the imagers in the imager array have different fields of view so that different scene sizes can be captured by the imagers.

18. The imaging device of claim 17, wherein the lens stacks of the at least two imagers have different focal lengths.

19. The imaging device of claim 17, wherein the light sensing elements in each of at least two imagers have different sizes.

20. The imaging device of claim 17, wherein the at least two imagers include different numbers of light sensing elements.

21. The imaging device of claim 1, further comprising a mechanical zoom mechanism within the lens stack of an imager for smoothly transitioning between different fields of view.

22. The imaging device of claim 1, wherein at least one of the imagers is designed to image light in the near-infrared spectrum.

23. The imaging device of claim 1, wherein at least one imager is configured to take a measurement of the visible light spectrum, and wherein the control circuitry is configured to use said measurement to control the imaging characteristics of the other imagers in said imaging device.

24. The imaging device of claim 1, wherein the imaging device includes at least two camera arrays.

25. The imaging device of claim 24, wherein the at least two camera arrays operate in conjunction during imaging to provide stereo images.

26. The imaging device of claim 24, wherein the at least two camera arrays are spatially separated one from the other in at least one plane.

27. The imaging device of claim 24, wherein each of the camera arrays comprise a different combination of imagers each combination of imagers having different imaging characteristics.

28. The imaging device of claim 1, wherein at least one camera array comprises an M×

N array of imagers, where at least one of N and M is greater than 2, and each of said imagers comprising at least two light sensing elements.

29. The imaging device of claim 1, wherein the control circuitry is configured to control the imaging characteristics of each imager.

30. The imaging device of claim 29, wherein the controllable imaging characteristics include exposure time, gain, and black level offset.

31. The imaging device of claim 1, wherein the control circuitry is configured to trigger each of said imagers in accordance with a desired sequence.

32. The imaging device of claim 31, wherein the control circuitry operates the imagers in a staggered sequence.

33. The imaging device of claim 1, wherein:

an initial master setting for the imaging characteristics of each imager is set by the control circuitry;

predefined deviations from the initial master settings are also set by the control circuitry; and

the deviations include functions selected from the group consisting of ISO settings, high dynamic range, gain settings, integration time settings, exposure settings, digital processing settings and combinations thereof.

34. The imaging device of claim 1, wherein the camera arrays are fabricated using techniques consisting of wafer level optics techniques, injection molding, glass molding and combinations thereof.

35. The imaging device of claim 1, wherein each of the plurality of imagers are optically separated from each other.

36. The imaging device of claim 35, wherein each of the imagers is optically separated from the other imagers by at least two opaque surfaces located in front of the light sensing elements and having openings arranged in axial alignment with the lens stack of the imager.

37. The imaging device of claim 35, wherein the imagers are optically separated from each other by opaque walls disposed at the boundaries between the imagers.

38. The imaging device of claim 37, wherein the opaque walls are cavities between the imagers filled with light blocking material.

39. The imaging device of claim 35, wherein:

the lens stack includes spacers; and

the spacers are constructed from an opaque material.

40. The imaging device of claim 35, wherein:

the lens stack includes spacers; and

the spacers are coated with an opaque material.

41. The imaging device of claim 1, wherein the control circuitry is configured to perform normalization of a captured low resolution image using a one dimensional space filling curve to describe a normalization plane.

42. The imaging device of claim 41, wherein the one dimensional space filling curve is expressed as the coefficients of a polynomial.

43. The imaging device of claim 1, wherein the control circuitry is configured to capture multiple low resolution images using different imagers having different exposure times.

44. The imaging device of claim 43, wherein the light sensing elements of at least one of the imagers used to capture an image are high dynamic range light sensing elements.

45. The imaging device of claim 44, wherein at least one of the imagers used to capture a low resolution image is configured to capture a near-IR low resolution image.

46. The imaging device of claim 1, wherein the super-resolution module is configured to select at least one distance as a focal plane and to apply blurring to pixels with depths that are not proximate a focal plane.

47. The imaging device of claim 1, wherein the super-resolution processing module is configured to perform temperature normalization with respect to the captured images.

48. The imaging device of claim 47, wherein:

the point spread functions of the lens stacks vary with temperature; and

the super-resolution processing module is configured to perform temperature normalization by selecting a point spread functions from temperature calibration data corresponding to the point spread functions of the lens stacks during the capture of the images.

49. The imaging device of claim 1, wherein the super-resolution processing module is configured to computationally correct color differences between the captured images.

50. The imaging device of claim 1, wherein the super-resolution processing module is configured to computationally correct geometric distortion differences between the captured images.

51. The imaging device of claim 1, wherein each of the plurality of imagers is of a given type with imagers of a same type sharing the same imaging characteristics and different types of imagers having different imaging characteristics.

52. The imaging device of claim 51, wherein the plurality of imagers are all of a same type that includes a Bayer spectral filter.

53. The imaging device of claim 51, wherein at least one of the plurality of imagers is of a different type from at least one other imager in the plurality of imagers.