CAPTURING PANORAMIC OR SEMI-PANORAMIC 3D SCENES

• US 20140104378A1
• Filed: 10/08/2013
• Published: 04/17/2014
• Est. Priority Date: 04/08/2011
• Status: Active Grant

• Current Assignee: Fraunhofer Gesellschaft Zur Foerderung Der Angewandten Forsching E.V.
• Original Assignee: Fraunhofer Gesellschaft Zur Foerderung Der Angewandten Forsching E.V.
• Inventors: KAUFF, Peter, EISERT, Peter, SCHUESSLER, Joachim, WEISSIG, Christian, FINN, Arne
• Granted Patent: US 9,462,184 B2
• Time in Patent Office: Days
• US Class Current: 348/38
• CPC Class Codes: G02B 13/06 Panoramic objectives; So-called "sky lenses" including panoramic objectives having reflecting surfaces
• CPC Class Codes: G02B 27/1066 for enhancing image performance, like resolution, pixel numbers, dual magnifications or dynamic range, by tiling, slicing or overlapping fields of view
• CPC Class Codes: G02B 27/143 using macroscopically faceted or segmented reflective surfaces
• CPC Class Codes: G03B 35/08 by simultaneous recording
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

A provision of high quality panoramic or semi-panoramic 3D scenes is achieved at reasonable efforts by arranging the plurality of camera pairs, each of which is associated with a respective one of mirror plane surfaces of a pyramid or clipped-pyramid shaped mirror in that same is directed towards the respective associated mirror plane surface, such that for each camera pair the virtual positions of pivot points of the cameras of the respective camera pair are offset from each other along a straight baseline which, in turn, is offset from the axis of the mirror, and a distance between virtual points of a left-hand channel camera of a first camera pair associated with a first mirror plane surface and a right-hand channel camera of a second camera pair associated with a second mirror plane surface positioned neighboring the first mirror plane surface, deviates from a length of the baselines of the camera pairs by less than 10% of the length of the baselines.

Claims

1. An apparatus for capturing panoramic or semi-panoramic 3D scenes, the apparatus comprising:

a mirror comprising a mirror surface composed of mirror plane surfaces, arranged like sides of a pyramid or clipped-pyramid with a polygonal base and an axis;

a plurality of camera pairs each of which is associated with a respective one of the mirror plane surfaces and directed towards the respective associated mirror plane surface, so that the plurality of camera pairs look into substantially radial directions via the respective associated mirror plane surface,

wherein, for each camera pair, virtual positions of pivot points of the cameras of the respective camera pair are offset from each other along a straight baseline which is offset from the axis, and

a distance between virtual points of a left-hand channel camera of a first camera pair associated with a first mirror plane surface and a right-hand channel camera of a second camera pair associated with a second mirror plane surface positioned neighboring the first mirror plane surface deviates from a length of the baselines of the camera pairs by less than 10% of the length of the baselines,

wherein center points of the baselines of all stereo camera pairs lie on a circle with radius d around axis within a deviation of 10% of d, and the length of the baselines varies within 10% of a mean baseline length B with

2. The apparatus according to claim 1, wherein the length of the baselines is a mean baseline length of the baselines the lengths of which vary within 10% of the mean baseline length.

3. The apparatus according to claim 1, wherein, for each camera pair, the virtual points of the pivot points of the cameras of the respective camera pair are positioned symmetrically left and right to a plane spanned by a normal vector of the respective associated mirror plane surface and the axis.

4. The apparatus according to claim 1, wherein centers of the baselines lie, within a deviation of 10% of a radius d, on a circle with radius d around axis.

5. The apparatus according to claim 1, wherein the virtual points of the pivot points of the cameras lie, within a deviation of 10% of a mean distance of centers of the baselines from the axis, on a circle with radius r=√

{square root over (d²+(B/2)²)}, where B is the mean length of the baselines.

6. The apparatus according to claim 1, wherein the virtual points of the pivot points of the cameras lie on a common plane perpendicular to the axis.

7. The apparatus according to claim 1, wherein an overlap between fields of view of the right-hand channel camera and the left-hand channel camera of the first and second camera pairs associated with the neighboring first and second mirror plane surfaces overlap each other by 5 to 20 pixel lines in average along a direction of the pixel lines.

8. The apparatus according to claim 1, wherein the cameras of the camera pairs comprise optical axis extending substantially in parallel to the axis.

9. The apparatus according to claim 1, wherein the cameras of each camera pair are rolled in opposite directions relative to each other around their optical axes.

10. The apparatus according to claim 1, wherein the cameras of each camera pair are registered to each other such that the fields of view thereof merely partially overlap each other along a lateral angle direction around the axis.

11. The apparatus according to claim 1, wherein the cameras of each camera pair are toed-in such that optical axes of the cameras intersect each other.

12. The apparatus according to claim 1, wherein the cameras of each camera pair are toed-in such that optical axes of the cameras intersect each other at the mirror plane surface with which the respective camera pair is associated.

13. The apparatus according to claim 1, wherein the cameras of the camera pairs are arranged so as to look into the substantially radial directions via the respective associated mirror plane surface in portrait format.

14. The apparatus according to claim 1, further comprising a processor configured to derive a panoramic or semi-panoramic picture or video from outputs of the camera pairs.

15. The apparatus according to claim 1, further comprising a processor configured to stitch pictures of left-hand channel cameras of the camera pairs on the one hand and pictures of right-hand channel cameras of the camera pairs on the other hand.

16. The apparatus according to claim 15, wherein the processor is configured to stitch the pictures of left-hand channel cameras of the camera pairs on the one hand and pictures of right-hand channel cameras of the camera pairs on the other hand such that, in the resulting left-hand and right-hand stitched views, only horizontal disparities exist between the left-hand and right-hand stitched views of each triple while vertical disparities and misalignments are removed.

17. The apparatus according to claim 15, wherein the processor is configured to stitch the pictures of left-hand channel cameras of the camera pairs on the one hand and pictures of right-hand channel cameras of the camera pairs on the other hand onto cylindrical surfaces around the axis in a convergence plane lying in infinity, and to rotate the two cylindrical surfaces of the left and right stereo channel in opposite directions around axis so as to shift the convergence plane from infinity to a desired depth level.

18. A method for capturing panoramic or semi-panoramic 3D scenes using a mirror comprising a mirror surface composed of mirror plane surfaces, arranged like sides of a pyramid or clipped-pyramid with a polygonal base and an axis, and a plurality of camera pairs each of which is associated with a respective one of the mirror plane surfaces and directed towards the respective associated mirror plane surface, so that the plurality of camera pairs look into substantially radial directions via the respective associated mirror plane surface, wherein, for each camera pair, virtual positions of pivot points of the cameras of the respective camera pair are offset from each other along a straight baseline which is offset from the axis, and a distance between virtual points of a left-hand channel camera of a first camera pair associated with a first mirror plane surface and a right-hand channel camera of a second camera pair associated with a second mirror plane surface positioned neighboring the first mirror plane surface deviates from a length of the baselines of the camera pairs by less than 10% of the length of the baselines,

wherein center points of the baselines of all stereo camera pairs lie on a circle with radius d around axis within a deviation of 10% of d, and the length of the baselines varies within 10% of a mean baseline length B with

19. The method according to claim 18, further comprising determining calibration data for stitching pictures of the left-hand and right-hand channel cameras, by capturing a test scenery by use of the cameras and

Separately stitching subsets of pictures of the left-hand channel cameras and subsets of pictures of the right-hand channel cameras;

Rectifying triples of the stitched pictures of the left-hand and right-hand cameras, overlapping each other, so that only horizontal disparities exist thereinafter, with the convergence plane being set to infinity; and

Mapping the triples onto locations of a cylindrical surface using feature-based stitching under maintaining the rectification.

20. A computer program comprising a program code for performing, when running on a computer, a method according to claim 19.