LIGHT-FIELD CAMERA

• US 20160248987A1
• Filed: 02/11/2016
• Published: 08/25/2016
• Est. Priority Date: 02/12/2015
• Status: Active Grant

• Current Assignee: Fraunhofer Gesellschaft Zur Foerderung Der Angewandten Forsching E.V.
• Original Assignee: Fraunhofer Gesellschaft Zur Foerderung Der Angewandten Forsching E.V.
• Inventors: ZILLY, Frederik, KEINERT, Joachim, ZIEGLER, Matthias, SCHOEBERL, Michael
• Granted Patent: US 10,511,787 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G02B 27/144 using partially transparent surfaces without spectral selectivity G02B27/147 takes precedence
• CPC Class Codes: G03B 35/10 having single camera with stereoscopic-base-defining system
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

The invention provides a light-field camera for capturing multiple views of a scene, representing samples of a light-field, having: a primary camera configured for capturing a primary digital two-dimensional image of the scene; a two-dimensional camera array having a plurality of secondary cameras, each of the secondary cameras being configured for capturing a secondary digital two-dimensional image of the scene to produce at least one set of secondary digital two-dimensional images of the scene; a semitransparent mirror arranged such that an incident light beam originating from the scene is split up in a first partial light beam, directed to the primary camera, and a second partial light beam, directed to the camera array; and a processing unit configured for receiving the primary digital two-dimensional image and the at least one set of secondary digital two-dimensional images and configured for computing depth information for the primary digital two-dimensional image or a digital two-dimensional image corresponding to the primary digital two-dimensional image based on the set of secondary digital two-dimensional images.

Claims

1. A light-field camera for capturing multiple views of a scene, the multiple views representing samples of a light-field, the light-field camera comprising:

a primary camera configured for capturing a primary digital two-dimensional image of the scene;

a two-dimensional camera array comprising a plurality of secondary cameras, wherein each of the secondary cameras is configured for capturing a secondary digital two-dimensional image of the scene in order to produce at least one set of secondary digital two-dimensional images of the scene;

a semitransparent mirror arranged in such way that an incident light beam originating from the scene is split up in a first partial light beam, which is directed to the primary camera, and a second partial light beam, which is directed to the camera array; and

a processing unit configured for receiving the primary digital two-dimensional image and the at least one set of secondary digital two-dimensional images and configured for computing depth information for the primary digital two-dimensional image or a digital two-dimensional image corresponding to the primary digital two-dimensional image based on the set of secondary digital two-dimensional images.

2. The light-field camera according to claim 1, wherein the primary camera fulfills at least one of the following conditions:

comprising a higher resolution than the secondary cameras, comprising a better quality of color reproduction than the secondary cameras, comprising a wider dynamic range than the secondary cameras, comprising a larger signal-to-noise ratio than the secondary cameras, comprising higher bit depth than the secondary cameras.

3. The light-field camera according to claim 1, wherein a primary objective of the primary camera comprises fewer distortions than secondary objectives of the secondary cameras.

4. The light-field camera according to claim 1, wherein the processing unit is configured for computing the depth information for the digital two-dimensional output image corresponding to the primary digital two-dimensional image solely based on the set of secondary digital two-dimensional images and on the position of the primary camera relative to the secondary cameras.

5. The light-field camera according to claim 1, wherein the secondary cameras comprise substantially parallel optical axes.

6. The light-field camera according to claim 1, wherein the secondary cameras comprise the same optical and/or electrical properties.

7. The light-field camera according to claim 1, wherein the secondary cameras are arranged in a lattice-like manner.

8. The light-field camera according to claim 1, wherein the primary camera, the camera array and the semitransparent mirror are mounted to a rig.

9. The light-field camera according to claim 7, wherein the rig comprises a mechanical adjuster for adjusting a crossing point of an optical axis of the primary camera and the optical axis of one of the secondary cameras.

10. The light-field camera according to claim 7, wherein the rig comprises a mechanical adjuster for compensating a difference of an optical path length from the scene to the primary camera and an optical path length from the scene to the camera array.

11. The light-field camera according to claim 7, wherein the rig comprises a mechanical adjuster for adjusting an angular orientation of the primary camera with respect to a rotation about a first rotation axis parallel to an optical axis of the primary camera relative to an angular orientation of the camera array with respect to a rotation about an rotation axis parallel to the optical axes of the secondary cameras.

12. The light-field camera according to claim 7, wherein the rig comprises a mechanical adjuster for compensating a difference of an angle from the optical axis of the primary camera to the first partial light beam and an angle from the second partial light beam to the optical axes of one of the secondary cameras.

13. The light-field camera according to claim 1, wherein the secondary cameras are configured for capturing a plurality of sets of secondary digital two-dimensional images while the primary camera captures one primary digital two-dimensional image, wherein each set of secondary digital two-dimensional images comprises different exposure settings than the other sets of secondary digital two-dimensional images, and wherein the processing unit is configured for computing the depth information based on the plurality of sets of secondary digital two-dimensional images.

14. The light-field camera according to claim 1, wherein the secondary cameras are combined into groups, wherein each group is configured to produce a subset of the set of secondary digital two-dimensional images of the scene, wherein each secondary camera comprises the same exposure settings as the other secondary cameras of the same group but different exposure settings than the secondary cameras of the other groups.

15. The light-field camera according to claim 1, wherein the processing unit comprises a rectification module configured for simultaneously rectifying of the secondary digital two-dimensional images of the set of secondary digital two-dimensional images and the primary digital two-dimensional image by using feature points or checkerboards in order to produce a set of rectified secondary digital two-dimensional images and a rectified primary digital two-dimensional image.

16. The light-field camera according to claim 14, wherein the processing unit comprises a depth information computing module configured for computing a depth information for each of the secondary cameras based on the set of rectified secondary digital two-dimensional images.

17. The light-field camera according to claim 15, wherein the processing unit comprises a depth warping module configured for warping each of the depth information to a position of the rectified primary digital two-dimensional image in order to produce the depth information for the primary digital two-dimensional image or the digital two-dimensional image corresponding to the primary digital two-dimensional image.

18. The light-field camera according to claim 1, wherein the processing unit comprises a rectification module configured for rectifying of the secondary digital two-dimensional images of the set of secondary digital two-dimensional images by using feature points or checkerboards in order to produce a set of rectified secondary digital two-dimensional images.

19. The light-field camera according to claim 17, wherein the processing unit comprises a depth information computing module configured for computing a depth information for each of the secondary cameras based on the set of rectified secondary digital two-dimensional images.

20. The light-field camera according to claim 18, wherein the processing unit comprises a depth warping module configured for warping each of the disparity maps to a position of the primary digital two-dimensional image in order to produce the depth information for the primary digital two-dimensional image or the digital two-dimensional image corresponding to the primary digital two-dimensional image.

21. The light-field camera according to claim 14, wherein the processing unit comprises a depth information computing module configured for computing a depth information for each of the secondary cameras and the depth information for the primary camera based on the set of rectified secondary digital two-dimensional images and the rectified primary image.

22. The light-field camera according to claim 14, wherein the light-field camera comprises a post-processing unit comprising a primary image warping module configured for warping the rectified primary digital two-dimensional image to a warped rectified primary digital two-dimensional image corresponding to a virtual camera position of the primary camera by using the depth information.

23. The light-field camera according to claim 21, wherein the post-processing unit comprises a secondary image warping module configured for warping each of the rectified secondary digital two-dimensional images into warped rectified secondary two-dimensional digital images corresponding to a position of the warped rectified primary two-dimensional image.

24. The light-field camera according to claim 22, wherein the post-processing unit comprises a disocclusion filling module configured for filling of disocclusions in the warped rectified primary digital two-dimensional image with information gathered from the warped rectified secondary digital two-dimensional images in order to produce an enhanced digital two-dimensional output image.

25. A method for capturing multiple views of a scene, the multiple views representing samples of a light-field, the method comprising:

capturing a primary digital two-dimensional image of the scene using a primary camera;

producing at least one set of secondary digital two-dimensional images of the scene using a two-dimensional camera array comprising a plurality of secondary cameras, wherein each of the secondary cameras is configured for capturing a secondary digital two-dimensional image of the scene;

providing a semitransparent mirror arranged in such way that an incident light beam originating from the scene is split up in a first partial light beam, which is directed to the primary camera, and a second partial light beam, which is directed to the camera array; and

receiving the primary digital two-dimensional image and the at least one set of secondary digital two-dimensional images and computing depth information for the primary digital two-dimensional image or a digital two-dimensional image corresponding to the primary digital two-dimensional image based on the set of secondary digital two-dimensional images using a processing unit.

26. A non-transitory digital storage medium having stored thereon a computer program for performing the method of claim 25 when said computer program is run by a computer.