Array cameras incorporating monolithic array camera modules with high MTF lens stacks for capture of images used in super-resolution processing
First Claim
1. An array camera, comprising:
- a camera array, where the camera array is a monolithic integrated module comprising;
a single semiconductor substrate on which light sensing elements for a plurality of focal planes are formed, where the light sensing elements of a given focal plane from the plurality of focal planes have a pixel pitch defining a Nyquist frequency;
optics including a plurality of lens elements forming a plurality of lens stacks, where each lens stack in the plurality of lens stacks;
forms an image on one of the plurality of focal planes;
has a field of view that is shifted with respect to the field-of-views of the other lens stacks and so that each shift includes a sub-pixel shifted view of the scene; and
has a modular transfer function (MTF) such that the lens stack optically resolves contrast at spatial frequencies larger than the Nyquist frequency (Ny); and
control circuitry that coordinates the independent capture of low resolution images of a scene by the focal planes;
a processor;
wherein each of the lens stacks in the plurality of lens stacks comprises a five-surface optical arrangement comprising;
a first lens element having a first convex proximal surface and a first concave distal surface, wherein the diameter of the first convex surface is larger than the diameter of the first concave surface;
a second lens element having a second concave proximal surface and a second convex distal surface, wherein the diameter of the second concave proximal surface is smaller than the diameter of the second convex surface; and
a third lens element having a third concave proximal surface and a third planar distal surface, wherein the diameter of the third concave proximal surface is larger than the diameters of any of the surfaces of the first and second lens elements;
wherein the first, second and thirds lens elements are arranged sequentially in optical alignment with an imager positioned at the distal end thereof; and
wherein software directs the processor to;
obtain a set of low resolution images from the camera array, where each of the low resolution images include aliasing patterns, where each of the low resolution images includes different aliasing patterns due to the different sub-pixel shifted views of the scene provided by the lens stacks;
determine disparity between pixels in the set of low resolution images to generate a depth map from a reference viewpoint, where the depth map indicates distances to surfaces of scene objects from the reference viewpoint;
synthesizing a high resolution image using the set of images and the depth map, where the spatial frequency at which the high resolution image displays contrast is greater than the Nyquist frequencies (Ny) of the plurality of focal planes and less than the spatial frequencies at which the plurality of lens stacks optically resolve contrast.
13 Assignments
0 Petitions
Accused Products
Abstract
A variety of optical arrangements and methods of modifying or enhancing the optical characteristics and functionality of these optical arrangements are provided. The optical arrangements being specifically designed to operate with camera arrays that incorporate an imaging device that is formed of a plurality of imagers that each include a plurality of pixels. The plurality of imagers include a first imager having a first imaging characteristics and a second imager having a second imaging characteristics. The images generated by the plurality of imagers are processed to obtain an enhanced image compared to images captured by the imagers. In many optical arrangements the MTF characteristics of the optics allow for contrast at spatial frequencies that are at least as great as the desired resolution of the high resolution images synthesized by the array camera, and significantly greater than the Nyquist frequency of the pixel pitch of the pixels on the focal plane, which in some cases may be 1.5, 2 or 3 times the Nyquist frequency.
-
Citations
17 Claims
-
1. An array camera, comprising:
-
a camera array, where the camera array is a monolithic integrated module comprising; a single semiconductor substrate on which light sensing elements for a plurality of focal planes are formed, where the light sensing elements of a given focal plane from the plurality of focal planes have a pixel pitch defining a Nyquist frequency; optics including a plurality of lens elements forming a plurality of lens stacks, where each lens stack in the plurality of lens stacks; forms an image on one of the plurality of focal planes; has a field of view that is shifted with respect to the field-of-views of the other lens stacks and so that each shift includes a sub-pixel shifted view of the scene; and has a modular transfer function (MTF) such that the lens stack optically resolves contrast at spatial frequencies larger than the Nyquist frequency (Ny); and control circuitry that coordinates the independent capture of low resolution images of a scene by the focal planes; a processor; wherein each of the lens stacks in the plurality of lens stacks comprises a five-surface optical arrangement comprising; a first lens element having a first convex proximal surface and a first concave distal surface, wherein the diameter of the first convex surface is larger than the diameter of the first concave surface; a second lens element having a second concave proximal surface and a second convex distal surface, wherein the diameter of the second concave proximal surface is smaller than the diameter of the second convex surface; and a third lens element having a third concave proximal surface and a third planar distal surface, wherein the diameter of the third concave proximal surface is larger than the diameters of any of the surfaces of the first and second lens elements; wherein the first, second and thirds lens elements are arranged sequentially in optical alignment with an imager positioned at the distal end thereof; and wherein software directs the processor to; obtain a set of low resolution images from the camera array, where each of the low resolution images include aliasing patterns, where each of the low resolution images includes different aliasing patterns due to the different sub-pixel shifted views of the scene provided by the lens stacks; determine disparity between pixels in the set of low resolution images to generate a depth map from a reference viewpoint, where the depth map indicates distances to surfaces of scene objects from the reference viewpoint; synthesizing a high resolution image using the set of images and the depth map, where the spatial frequency at which the high resolution image displays contrast is greater than the Nyquist frequencies (Ny) of the plurality of focal planes and less than the spatial frequencies at which the plurality of lens stacks optically resolve contrast. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)
-
-
15. An array camera, comprising:
-
a camera array, where the camera array is a monolithic integrated module comprising; a single semiconductor substrate on which light sensing elements for a plurality of focal planes are formed, where the light sensing elements of a given focal plane from the plurality of focal planes have a pixel pitch defining a Nyquist frequency; optics including a plurality of lens elements forming a plurality of lens stacks, where each lens stack in the plurality of lens stacks; forms an image on one of the plurality of focal planes; has a field of view that is shifted with respect to the field-of-views of the other lens stacks and so that each shift includes a sub-pixel shifted view of the scene; and has a modular transfer function (MTF) such that the lens stack optically resolves contrast at spatial frequencies larger than the Nyquist frequency (Ny); and control circuitry that coordinates the independent capture of low resolution images of a scene by the focal planes; wherein each of the lens stacks in the plurality of lens stacks comprises a five-surface optical arrangement comprising; a first lens element having a first convex proximal surface and a first concave distal surface, wherein the diameter of the first convex surface is larger than the diameter of the first concave surface; a second lens element having a second concave proximal surface and a second convex distal surface, wherein the diameter of the second concave proximal surface is smaller than the diameter of the second convex surface; and a third lens element having a third concave proximal surface and a third planar distal surface, wherein the diameter of the third concave proximal surface is larger than the diameters of any of the surfaces of the first and second lens elements; wherein the first second and thirds lens elements are arranged sequentially in optical alignment with an imager positioned at the distal end thereof; and a processor; wherein software directs the processor to; obtain a set of low resolution images from the camera array, where each of the low resolution images include aliasing patterns, where each of the low resolution images includes different aliasing patterns due to the different sub-pixel shifted views of the scene provided by the lens stacks; determine disparity between pixels in the set of low resolution images to generate a depth map from a reference viewpoint, where the depth map indicates distances to surfaces of scene objects from the reference viewpoint; synthesizing a high resolution image using the set of images and the depth map by; determining scene dependent geometric corrections to apply to the pixels from each of the images within the set of low resolution images to eliminate disparity; fusing the set of low resolution images using he scene dependent geometric corrections; and performing super resolution processing to reconstruct the high resolution image using the fused image, the scene dependent geometric corrections, and the set of low resolution images; wherein the spatial frequency at which the high resolution image displays contrast is greater than the camera Nyquist frequencies (Ny) of the plurality of focal planes and less than the spatial frequencies at which the plurality of lens stacks optically resolve contrast; wherein each of the cameras formed by the lens stack and focal planes include one of a plurality of different types of filer; and wherein cameras that include different types of filter operate with different operating parameters.
-
-
16. An array camera, comprising:
-
a camera array, where the camera array is a monolithic integrated module comprising; a single semiconductor substrate on which light sensing elements for a plurality of focal planes are formed, where the light sensing elements of a given focal plane from the plurality of focal planes have a pixel pitch defining a Nyquist frequency; optics including a plurality of lens elements forming a plurality of lens stacks, where each lens stack in the plurality of lens stacks; forms an image on one of the plurality of focal planes; has a field of view that is shifted with respect to the field-of-views of the other lens stacks and so that each shift includes a sub-pixel shifted view of the scene; and has a modular transfer function (MTF) such that the lens stack optically resolves contrast at spatial frequencies larger than the Nyquist frequency (Ny); and control circuitry that coordinates the independent capture of low resolution images of a scene by the focal planes; wherein the plurality of lens stacks are part of a three-element monolithic lens optical arrangement comprising; a first lens element having a first convex proximal surface and a first plano distal surface; a second lens element having a second concave proximal surface and a second convex distal surface; a third menisci lens element having a third concave proximal surface and a third convex distal surface; and at least one aperture disposed on the first plano distal surface; wherein the first, second and third lens elements are arranged sequentially in optical alignment with the aperture stop and an imager; and a processor; wherein software directs the processor to; obtain a set of low resolution images from the camera array, where each of the low resolution images include aliasing patterns, where each of the low resolution images includes different aliasing patterns due to the different sub-pixel shifted views of the scene provided by the lens stacks; determine disparity between pixels in the set of low resolution images to generate a depth map from a reference viewpoint, where the depth map indicates distances to surfaces of scene objects from the reference viewpoint; synthesizing a high resolution image using the set of images and the depth map by; determining scene dependent geometric corrections to apply to the pixels from each of the images within the set of low resolution images to eliminate disparity; fusing the set of low resolution images using the scene dependent geometric corrections; and performing super resolution processing to reconstruct the high resolution image using the fused image, the scene dependent geometric corrections, and the set of low resolution images; wherein the spatial frequency at which the high resolution image displays contrast is greater than the camera Nyquist frequencies (Ny) of the plurality of focal planes and less than the spatial frequencies at which the plurality of lens stacks optically resolve contrast; wherein each of the cameras formed by the lens stack and focal planes include one of a plurality of different types of filer; and wherein cameras that include different types of filter operate with different operating parameters.
-
-
17. An array camera, comprising:
-
a camera array, where the camera array is a monolithic integrated module comprising; a single semiconductor substrate on which light sensing elements for a plurality of focal planes are formed, where the light sensing elements of a given focal plane from the plurality of focal planes have a pixel pitch defining a Nyquist frequency; optics including a plurality of lens elements forming a plurality of lens stacks, where each lens stack in the plurality of lens stacks; forms an image on one of the plurality of focal planes; has a field of view that is shifted with respect to the field-of-views of the other lens stacks and so that each shift includes a sub-pixel shifted view of the scene; and has a modular transfer function (MTF) such that the lens stack optically resolves contrast at spatial frequencies larger than the Nyquist frequency (Ny); and control circuitry that coordinates the independent capture of low resolution images of a scene by the focal planes; a processor; wherein the plurality of lens stacks are part of a three-element monolithic lens optical arrangement comprising; a first lens element having a first convex proximal surface and a first plano distal surface; a second lens element having a second concave proximal surface and a second convex distal surface; a third menisci lens element having a third concave proximal surface and a third convex distal surface; and at least one aperture disposed on the first plano distal surface; wherein the first, second and third lens elements are arranged sequentially in optical alignment with the aperture stop and an imager; and wherein software directs the processor to; obtain a set of low resolution images from the camera array, where each of the low resolution images include aliasing patterns, where each of the low resolution images includes different aliasing patterns due to the different sub-pixel shifted views of the scene provided by the lens stacks; determine disparity between pixels in the set of low resolution images to generate a depth map from a reference viewpoint, where the depth map indicates distances to surfaces of scene objects from the reference viewpoint; synthesizing a high resolution image using the set of images and the depth map, where the spatial frequency at which the high resolution image displays contrast is greater than the Nyquist frequencies (Ny) of the plurality of focal planes and less than the spatial frequencies at which the plurality of lens stacks optically resolve contrast.
-
Specification