Capturing and processing of images using monolithic camera array with heterogeneous imagers

• US 8,861,089 B2
• Filed: 07/22/2013
• Issued: 10/14/2014
• Est. Priority Date: 11/20/2009
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Pelican Imaging Corporation
• Inventors: Duparre, Jacques
• Primary Examiner(s): PINKNEY, DAWAYNE
• Application Number:
  US13/948,054
• Publication Number:
  US 20130308197A1
• Time in Patent Office: 449 Days
• Field of Search: 359618-621, 359625-626, 359/443, 359454-455, 264/1.1, 264/1.32, 264/2.7
• US Class Current: 359/618
• CPC Class Codes: G02B 13/001 Miniaturised objectives for electronic devices, e.g. portable telephones, webcams, PDAs, small digital cameras G02B23/243 takes precedence
• CPC Class Codes: G02B 3/0056 arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085
• CPC Class Codes: G02B 3/0062 Stacked lens arrays, i.e. refractive surfaces arranged in at least two planes, without structurally separate optical elements in-between
• CPC Class Codes: G02B 5/201 in the form of arrays
• CPC Class Codes: H01L 27/14621 Colour filter arrangements
• CPC Class Codes: H01L 27/14625 Optical elements or arrangements associated with the device
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/16 Optical arrangements associated therewith, e.g. for beam-splitting or for colour correction
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/41 Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors
• CPC Class Codes: H04N 25/76 Addressed sensors, e.g. MOS or CMOS sensors

Abstract

Systems and methods for implementing array cameras configured to perform super-resolution processing to generate higher resolution super-resolved images using a plurality of captured images and lens stack arrays that can be utilized in array cameras are disclosed. Lens stack arrays in accordance with many embodiments of the invention include lens elements formed on substrates separated by spacers, where the lens elements, substrates and spacers are configured to form a plurality of optical channels, at least one aperture located within each optical channel, at least one spectral filter located within each optical channel, where each spectral filter is configured to pass a specific spectral band of light, and light blocking materials located within the lens stack array to optically isolate the optical channels.

Claims

1. A camera array, comprising:

a lens stack array, comprising;

lens elements arranged to form a plurality of optical channels, where the optical channels are configured to form an image of an object space within a scene from slightly different viewpoints;

at least one aperture located within each optical channel;

at least one spectral filter located within each optical channel, where each spectral filter is configured to pass a specific spectral band of light; and

light blocking materials located within the lens stack array to optically isolate the optical channels; and

an imager array, comprising;

a plurality of imagers, where each imager comprises at least two pixels; and

circuitry for controlling imaging parameters that is configured to trigger each imager independently;

wherein the plurality of optical channels and the plurality of imagers form a plurality of cameras configured to capture different low resolution images representing slightly different viewpoints of the same scene that provide non-redundant information about the scene;

wherein the low resolution images captured by the plurality of cameras include different occlusions sets, where the occlusion set of a given camera is the portion of a scene visible to a first of the plurality of cameras that is occluded from the view of the given camera;

wherein the shape of at least one lens element forming a first optical channel for a camera of a first type that includes a spectral filter configured to pass a first spectral band of light differs from the shape of at least one lens element forming a second optical channel for a camera of a second type that includes a spectral filter configured to pass a second spectral band of light; and

wherein the plurality of cameras includes at least one camera of each type that captures pixels around an edge of a foreground object that is visible to the first of the plurality of cameras and is in the occlusion set of another of the plurality of cameras.

2. The camera array of claim 1, wherein the light blocking materials in the lens stack array are selected from the group consisting of opaque materials, reflective materials and combinations thereof.

3. The camera array of claim 1, wherein each spectral filter is selected from the group consisting of an organic color filter, an absorptive material, a dielectric coating, an interference filter, a multilayer coating, and combinations thereof.

4. The camera array of claim 1, further comprising at least one polarizing filter located within each optical channel.

5. The camera array of claim 1, wherein the prescription of at least one surface of a lens within each optical channel is a function of the specific spectral band of light passed by the spectral filter within the optical channel.

6. The camera array of claim 5, wherein the back focal lengths of each optical channel in the lens stack array are the same irrespective of the spectral band of light passed by the spectral filter within the optical channel.

7. The camera array of claim 1, wherein a combination of high and low Abbe number materials is used in the construction of each of the lens elements in an optical channel to reduce chromatic aberrations.

8. The camera array of claim 1, wherein each spectral filter is located within each optical channel so that the spectral filter is proximate the aperture stop.

9. The camera array of claim 8, wherein each aperture stop is formed by a light blocking material selected from the group consisting of metal materials, oxide materials, black particle filled photoresists and combinations thereof.

10. The camera array of claim 1, wherein:

at least one lens surface of a lens in each optical channel differs based upon the specific spectral band of light passed by the spectral filter within the optical channel; and

each lens surface is selected from the group consisting of diffractive, Fresnel, refractive and combinations thereof.

11. The camera array of claim 10, wherein the radii of curvature of the lens surfaces differ based upon the specific spectral band of light passed by the spectral filter within the optical channel.

12. The camera array of claim 10, wherein each of the optical channels have the same back focal length.

13. The camera array of claim 1, wherein at least one of the lens elements in an optical channel is a negative lens element, and the negative lens element is proximate the image formed by the optical channel.

14. The camera array of claim 1, wherein at least two of the optical channels have different focal lengths.

15. The camera array of claim 1, further comprising a mechanical zoom mechanism within an optical channel configured to smoothly transition between different fields of view.

16. The camera array of claim 1, wherein the lens stack array comprises an M×

N array of optical channels, where at least one of N and M is greater than 2.

17. The camera array of claim 1, wherein the lens stack array is fabricated using techniques consisting of wafer level optics techniques, injection molding, glass molding and combinations thereof.

18. The camera array of claim 1, wherein the light blocking materials located within the lens stack array to optically isolate the optical channels comprise at least two opaque surfaces located on a substrate within an optical channel, where the two opaque surfaces have openings arranged in axial alignment with the optical channel.

19. The camera array of claim 1, where the sampling diversity of the low resolution images is dependent upon distance to the part of the object space that is sampled.

20. The camera array of claim 1, wherein:

the lens elements in the lens stack array are formed on substrates separated by spacers; and

at least one spectral filter is located within a space defined by the substrates separated by spacers.

21. The camera array of claim 10, wherein at least one spectral filter is disposed on the surface of a lens element within the lens stack array.

22. The camera array of claim 1, wherein at least one spectral filter is located on the surface of the imager array.

23. The camera array of claim 1, wherein at least one camera is a polychromatic camera.

24. The camera array of claim 1, wherein at least one camera is a near-IR camera.

25. The camera array of claim 1, wherein the at least one spectral filter located within each optical channel forms at least a 3×

3 array of cameras comprising;

a camera having a green filter at the center of a 3×

3 configuration of cameras;

two cameras having red filters located on opposite sides of the 3×

3 configuration of cameras;

two cameras having blue filters located on opposite sides of the 3×

3 configuration of cameras; and

four cameras having green filters surrounding the central camera having a green filter.

26. The camera array of claim 1, wherein the camera array comprises an array of between 2×

2 and 6×

6 cameras.

27. The camera array of claim 26, wherein the camera array comprises a 3×

3 array of cameras.

28. The camera array of claim 26, wherein the camera array comprises a 4×

4 array of cameras.

29. The camera array of claim 26, wherein the camera array comprises a 5×

5 array of cameras.

30. The camera array of claim 1, wherein the camera array comprises a one-dimensional array of cameras.

31. The camera array of claim 30, wherein the camera array comprises a 1×

10 array of cameras.

32. The camera array of claim 1, wherein the pixels captured by the at least one camera of each type that captures pixels around an edge of a foreground object that is visible to the first of the plurality of cameras and is in the occlusion set of another of the plurality of cameras enable reconstruction of color information around the edge of the foreground object that is visible to the first of the plurality of cameras.