Multi-camera system using folded optics free from parallax artifacts

• US 9,386,222 B2
• Filed: 01/30/2015
• Issued: 07/05/2016
• Est. Priority Date: 06/20/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Georgiev, Todor Georgiev, Osborne, Thomas Wesley, Goma, Sergiu Radu
• Primary Examiner(s): Jerabek, Kelly L
• Application Number:
  US14/611,045
• Publication Number:
  US 20150373263A1
• Time in Patent Office: 522 Days
• Field of Search: 348/36, 348/38, 348/39, 348/143, 348/218.1, 348/239, 348/262
• US Class Current: 1/1
• CPC Class Codes: G02B 17/023 for extending or folding an optical path, e.g. delay lines
• CPC Class Codes: G02B 27/1066 for enhancing image performance, like resolution, pixel numbers, dual magnifications or dynamic range, by tiling, slicing or overlapping fields of view
• CPC Class Codes: G02B 27/40 Optical focusing aids
• CPC Class Codes: G03B 17/00 Details of cameras or camera bodies; Accessories therefor lens hoods or caps G03B11/04
• CPC Class Codes: G03B 17/17 with reflectors arranged in beam forming the photographic image, e.g. for reducing dimensions of camera
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Aspects relate to an array camera exhibiting little or no parallax artifacts in captured images. For example, the planes of the central mirror surfaces of the array camera can be located at a midpoint along, and orthogonally to, a line between the corresponding camera location and the virtual camera location. Accordingly, the cones of all of the cameras in the array appear as if coming from the virtual camera location after folding by the mirrors. Each sensor in the array “sees” a portion of the image scene using a corresponding facet of the central mirror prism, and accordingly each individual sensor/mirror pair represents only a sub-aperture of the total array camera. The complete array camera has a synthetic aperture generated based on the sum of all individual aperture rays.

Claims

1. A system for capturing a target image of a scene, the system comprising:

an array of a plurality of cameras having a virtual center of projection, each of the plurality of cameras configured to capture one of a plurality of portions of the target image of the scene;

each of the plurality of cameras comprising;

an image sensor;

a lens assembly comprising at least one lens, the lens assembly having a center of projection, the lens assembly positioned to focus light on the image sensor, the center of projection of the lens assembly located along a line passing through the virtual center of projection; and

a mirror positioned to reflect the light to the lens assembly, the mirror further positioned on a mirror plane, the mirror plane positioned to intersect a midpoint along the line passing through the virtual center of projection and the center of projection of the lens assembly, the mirror plane further positioned at an angle orthogonal to the line; and

a surface positioned above the array, the surface positioned to allow light representing the scene to pass through an aperture to the mirror of each of the plurality of cameras.

2. The system of claim 1, further comprising a central reflective element having a plurality of primary light re-directing surfaces configured to split the light into the plurality of portions, the mirror of each of the plurality of cameras forming one of the primary light re-directing surfaces.

3. The system of claim 2, further comprising a secondary light re-directing surface associated with each of the plurality of cameras, the secondary light re-directing surface positioned to direct light from the lens assembly onto the image sensor.

4. The system of claim 3, further comprising a substrate, wherein the image sensor for each of the plurality of cameras is positioned on or within the substrate.

5. The system of claim 1, wherein the central reflective element comprises an apex formed by an intersection of each of the plurality of primary light re-directing surfaces, and wherein an optical axis of each of the plurality of cameras passes through the apex.

6. The system of claim 5, wherein the apex and the virtual center of projection are located on a virtual optical axis passing through the apex and the virtual center of projection such that the virtual optical axis forms an optical axis of the array of the plurality of cameras.

7. The system of claim 5, wherein the surface is positioned orthogonal to the virtual optical axis at or above the apex of the central reflective element, the system further comprising a lower surface positioned substantially parallel to the upper surface and positioned at or below a lower surface of the central reflective element.

8. The system of claim 7, wherein each of the plurality of cameras is positioned between the upper surface and the lower surface.

9. The system of claim 8, wherein a distance between the upper surface and the lower surface is approximately 4.5 mm or less.

10. The system of claim 1, further comprising a camera housing comprising the surface having the aperture as an upper surface thereof, wherein the aperture is positioned to allow the light representing the scene to pass through the aperture to the central reflective element.

11. The system of claim 1, further comprising a processor configured to receive image data comprising the plurality of portions of the scene from each of the plurality of cameras and to perform a stitching operation to generate the target image.

12. The system of claim 11, wherein the processor is further configured to perform a projective transform on the image data based at least partly a geometric relationship between the virtual center of projection and the center of projection of the lens assembly of each of the plurality of cameras and the location of the mirror for each of the plurality of cameras within a corresponding mirror plane.

13. A method of manufacturing an array camera substantially free of parallax artifacts in a captured image, comprising:

positioning a plurality of cameras in an array, each of the plurality of cameras configured to capture one of a plurality of portions of a target image scene, the plurality of cameras positioned to each capture image data from a location of a virtual camera having a virtual center of projection, each of the plurality of cameras comprising;

a lens assembly comprising at least one lens,

a camera center of projection having a location determined at least partly by optics of the lens assembly, and

a sensor positioned to receive light from the lens assembly; and

providing, for each of the plurality of cameras, a mirror positioned on a mirror plane, the mirror plane positioned to intersect a midpoint along a line passing through the virtual center of projection the camera center of projection, the mirror plane further positioned at an angle orthogonal to the line.

14. The method of claim 13, further comprising providing the mirror for each of the plurality of cameras as a facet of a central reflective element, the plurality of cameras positioned around the central reflective element.

15. The method of claim 14, wherein the mirror for each of the plurality of cameras intersects at an apex of the central reflective element, and wherein the method further comprises positioning each of the plurality of cameras such that an optical axis of each of the plurality of cameras intersects with the apex.

16. The method of claim 13, further comprising providing a secondary light re-directing surface for each of the plurality of cameras.

17. The method of claim 16, further comprising positioning the secondary light re-directing surface between the lens assembly and the sensor to direct light received from the lens assembly onto the sensor.

18. The method of claim 17, further comprising providing a substrate having at least one aperture positioned to allow the light to pass through the aperture to the mirror for each of the plurality of cameras.

19. The method of claim 18, further comprising mounting the sensor for each of the plurality of cameras on or within the substrate such that all sensors are positioned within the same plane.

20. The method of claim 13, further comprising providing a processor in electronic communication with the sensor of each of the plurality of cameras, the processor configured to receive the plurality of portions of a target image scene and generate a complete image of the target image scene.

21. An apparatus for capturing a target image of a scene, the apparatus comprising:

an array of a plurality of cameras having a virtual center of projection, each of the plurality of cameras configured to capture one of a plurality of portions of the target image of the scene; and

for each of the plurality of cameras;

means for capturing an image;

means for focusing light having a center of projection located along a line passing through the virtual center of projection; and

means for redirecting light located on a primary light re-directing plane positioned to intersect a midpoint along the line passing through the virtual center of projection and the means for capturing an image, the primary light re-directing plane further positioned at an angle orthogonal to the line.

22. The apparatus of claim 21, wherein the means for redirecting light comprises one of a reflective surface positioned within the primary light folding plane and a refractive prism having a facet positioned within the primary light folding plane.

23. The apparatus of claim 21, wherein the means for focusing light comprises a lens assembly including one or more lenses.

24. The apparatus of claim 21, wherein each of the plurality of cameras comprises an additional means for re-directing light positioned to direct light received from the light focusing means onto the image capture means.

25. The apparatus of claim 21, further comprising means for combining the plurality of portions into the target image.

26. The apparatus of claim 25, further comprising means for compensating for tilt artifacts between the plurality of portions of the target image.

27. A system for capturing a target image of a scene, the system comprising:

an array of a plurality of cameras having a virtual center of projection, each of the plurality of cameras configured to capture one of a plurality of portions of a target image of a scene;

each of the plurality of cameras comprising;

an image sensor,

a lens assembly comprising at least one lens, the lens assembly having a center of projection, the lens assembly positioned to focus light on the image sensor, the center of projection of the lens assembly located along a line passing through the virtual center of projection, and

a mirror positioned to reflect the light to the lens assembly, the mirror further positioned on a mirror plane, the mirror plane positioned to intersect a point along the line passing through the virtual center of projection;

a central reflective element having;

a plurality of primary light re-directing surfaces configured to split the light into the plurality of portions, the mirror of each of the plurality of cameras forming one of the primary light re-directing surfaces, and

an apex formed by an intersection of each of the plurality of primary light re-directing surfaces, an optical axis of each of the plurality of cameras passing through the apex; and

wherein the apex and the virtual center of projection are located on a virtual optical axis passing through the apex and the virtual center of projection such that the virtual optical axis forms an optical axis of the array of the plurality of cameras.

28. The system of claim 27, wherein the mirror plane is positioned to intersect a midpoint along the line passing through the virtual center of projection and an associated one of the plurality of cameras, the mirror plane further and positioned at an angle orthogonal to the line.

29. The system of claim 27, further comprising a camera housing comprising:

an upper surface having an aperture positioned to allow the light representing the scene to pass through the aperture to the mirror of each of the plurality of cameras; and

a lower surface positioned substantially parallel to the upper surface.

30. The system of claim 29, wherein each of the plurality of cameras is positioned between the upper surface and the lower surface, and wherein a distance between the upper surface and the lower surface is approximately 4.5 mm or less.