Multi-camera system using folded optics free from parallax artifacts

• US 9,733,458 B2
• Filed: 05/24/2016
• Issued: 08/15/2017
• Est. Priority Date: 06/20/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Georgiev, Todor Georgiev, Osborne, Thomas Wesley, Goma, Sergiu Radu
• Primary Examiner(s): Jerabek, Kelly L
• Application Number:
  US15/163,091
• Publication Number:
  US 20160286121A1
• Time in Patent Office: 448 Days
• Field of Search: 348 36, 348 38- 39, 348143, 3482181, 348239, 348262
• CPC Class Codes: G02B 17/023 for extending or folding an optical path, e.g. delay lines
• CPC Class Codes: G02B 27/1066 for enhancing image performance, like resolution, pixel numbers, dual magnifications or dynamic range, by tiling, slicing or overlapping fields of view
• CPC Class Codes: G02B 27/40 Optical focusing aids
• CPC Class Codes: G03B 17/00 Details of cameras or camera bodies; Accessories therefor lens hoods or caps G03B11/04
• CPC Class Codes: G03B 17/17 with reflectors arranged in beam forming the photographic image, e.g. for reducing dimensions of camera
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Aspects relate to an array camera exhibiting little or no parallax artifacts in captured images. For example, the planes of the central mirror surfaces of the array camera can be located at a midpoint along, and orthogonally to, a line between the corresponding camera location and the virtual camera location. Accordingly, the cones of all of the cameras in the array appear as if coming from the virtual camera location after folding by the mirrors. Each sensor in the array “sees” a portion of the image scene using a corresponding facet of the central mirror prism, and accordingly each individual sensor/mirror pair represents only a sub-aperture of the total array camera. The complete array camera has a synthetic aperture generated based on the sum of all individual aperture rays.

Claims

1. A system for capturing an image of a target scene, the system comprising:

an array of a plurality of cameras positioned to capture image data representing a plurality of portions of the target scene, the array having a virtual center of projection;

each of the plurality of cameras positioned to capture a portion of the image data representing a different one of the plurality of portions of the target scene, and each of the plurality of cameras comprising

an image sensor,

a lens assembly comprising at least one lens, the lens assembly having a center of projection, and

a first reflective surface positioned to reflect incoming light representing one of the plurality of portions of the target scene toward the lens assembly, the first reflective surface provided in a plane positioned

at a midpoint along a line connecting the center of projection of the lens assembly and the virtual center of projection, and

at an angle orthogonal to the line;

a surface positioned between the array and the target scene, the surface including an aperture positioned to allow light representing the target scene to pass to the first reflective surface of each of the plurality of cameras;

a processor; and

a memory storing instructions that configure the processor to generate the image of the target scene based at least partly on the image data comprising each of the plurality of portions of the target scene;

wherein the image sensor of each of the plurality of cameras is positioned within a common plane.

2. The system of claim 1, each of the plurality of cameras further comprising a second reflective surface positioned to reflect light received from the lens assembly onto the image sensor, wherein the lens assembly is positioned between the first and second reflective surfaces.

3. The system of claim 1, further comprising a substrate having the image sensor of each of the plurality of cameras mounted to the substrate.

4. The system of claim 3, wherein the surface comprises the substrate, each of the plurality of cameras comprising a refractive prism comprising the first reflective surface and an input surface, the input surface of the refractive prism positioned in the aperture.

5. The system of claim 3, further comprising a central reflective element mounted to the substrate in the path of light representing the target scene passing through the aperture of the surface, the central reflective element having a plurality of facets configured to split the light representing the target scene into the plurality of portions, the first reflective surface of each of the plurality of cameras forming one of the plurality of facets.

6. The system of claim 5, wherein the central reflective element comprises an apex formed by an intersection of each of the first reflective surfaces, and wherein an optical axis of each of the plurality of cameras passes through the apex.

7. The system of claim 6, wherein the array has a virtual optical axis, and wherein the apex and the virtual center of projection are located on the virtual optical axis.

8. The system of claim 1, further comprising a camera housing comprising:

the surface having the aperture as an upper surface thereof; and

a lower surface positioned substantially parallel to the upper surface.

9. The system of claim 8, wherein each of the plurality of cameras is positioned between the upper surface and the lower surface.

10. The system of claim 1, wherein the instructions stored in the memory further configure the processor to perform a projective transform on the image data based at least partly on a geometric relationship between the virtual center of projection and the center of projection of the lens assembly of each of the plurality of cameras and the location of the mirror for each of the plurality of cameras within a corresponding mirror plane.

11. An apparatus for capturing an image of a target scene, the apparatus comprising:

an array of a plurality of cameras having a virtual center of projection, the array positioned to capture image data representing a plurality of portions of the target scene;

each of the plurality of cameras positioned to capture a portion of the image data representing a different one of the plurality of portions of the target scene, each of the plurality of cameras comprising

image sensing means for generating the image data in response to received light,

means for focusing light, the means for focusing having a center of projection, and

first light folding means positioned to redirect incoming light representing one of the plurality of portions of the target scene toward the means for focusing, the first light folding means provided in a plane positioned

at a midpoint along a line connecting the center of projection of the means for focusing and the virtual center of projection, and

at an angle orthogonal to the line;

housing means positioned between the array and the target scene, the housing means including an aperture positioned to allow light representing the target scene to be received by the first light folding means of each of the plurality of cameras; and

means for mounting the image sensing means of each of the plurality of cameras within a common plane.

12. The apparatus of claim 11, each of the plurality of cameras further comprising second light folding means positioned to redirect light received from the means for focusing onto the image sensing means, wherein the lens assembly is positioned between the first and second reflective surfaces.

13. The apparatus of claim 12, further comprising means for moving at least one of the first and second light folding means to change a focal length between the means for focusing and the image sensing means.

14. The apparatus of claim 11, further comprising means for combining the image data representing the plurality of portions of the target scene into the image.

15. The apparatus of claim 11, further comprising means for moving the means for focusing to change a focal length between the means for focusing and the image sensing means.

16. The apparatus of claim 11, further comprising means for adjusting the focus of each of the plurality of cameras together, the means for adjusting positioned between the first light folding means of each of the plurality of cameras and the target scene.

17. A method of capturing an image of a target scene such that the image is substantially free of parallax and tilt artifacts, comprising:

receiving light representing the target scene through an aperture to a plurality of mirrors;

splitting the light representing the target scene into a plurality of portions using the plurality of mirrors;

capturing image data representing the plurality of portions of the target scene by a corresponding plurality of cameras, each of the plurality of cameras positioned to capture a portion of the image data representing a different one of the plurality of portions from a location of a virtual camera having a virtual center of projection, each of the plurality of cameras comprising

a mirror of the plurality of mirrors,

a lens assembly comprising at least one lens, the lens assembly positioned to receive light representing one of the plurality of portions from the mirror,

a center of projection of the lens assembly, and

a sensor positioned to receive the light from the lens assembly,

wherein the mirror is provided in a plane positioned

at a midpoint along a line connecting the center of projection of the lens assembly and the virtual center of projection, and

at an angle orthogonal to the line; and

assembling the image data representing the plurality of portions of the target scene into the image.

18. The method of claim 17, further comprising applying projective transform to the image data representing the plurality of portions of the target scene to change an effective tilt of each of the plurality of cameras.

19. The method of claim 17, further comprising, for each of the plurality of cameras, redirecting the light received from the lens assembly onto the image sensor using a secondary light re-directing surface.

20. The method of claim 17, wherein assembling image data representing the plurality of portions of the target scene into the image comprises stitching portions of the image data representing each of the plurality of portions into a single image and cropping the single image to a specified aspect ratio.