APPARATUS FOR SYNTHESIZING THREE-DIMENSIONAL IMAGES TO VISUALIZE SURROUNDINGS OF VEHICLE AND METHOD THEREOF

• US 20140125774A1
• Filed: 06/23/2011
• Published: 05/08/2014
• Est. Priority Date: 06/21/2011
• Status: Active Grant

• Current Assignee: NC& Company Limited
• Original Assignee: Vadas, Ltd.
• Inventors: Lee, Jun-Seok, Jeon, Byung-Chan, Park, Gyu-Ro, Ryu, Sang-Jin, Song, Byeong-Moon
• Granted Patent: US 9,451,236 B2
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 13/20 Image signal generators
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"

Abstract

The present invention relates to three-dimensional visualization of the surrounding images of a vehicle, and comprises the steps of: enabling a plurality of wide angle cameras provided at a vehicle to receive a plurality of photographed images for reference patterns formed on the ground; extracting feature points from the photographed reference patterns and estimating a relative location and an installation angle of each camera using the known physical location information of the extracted feature points; obtaining optical parameters comprising an optical center of a lens for correcting lens distortion using the photographed images and mapping each image on a surface of a Gaussian sphere using the obtained optical center; changing an angle and distance such that the Gaussian sphere and the real reference patterns correspond to each other using the relative location and the installation angle of the estimated camera and arranging the images in a three-dimensional virtual space; and obtaining a three-dimensional single image by mapping each image arranged in the three-dimensional virtual space to an inner surface of the three-dimensional sphere corresponding to one large sphere.

Claims

1. An apparatus for synthesizing three-dimensional (3D) images, comprising:

an image input/output unit configured to receive photographed images of a reference pattern formed on a ground and a non-ground plane from a plurality of wide angle cameras mounted on a vehicle, and to output the photographed images;

an image arrangement estimation unit configured to extract feature points from the reference patterns of the image input from the image input/output unit, and to estimate a relative location and installation angle of each of the cameras using known physical location information of the extracted feature points;

a sphere mapping unit configured to obtain optical parameters for correction of lens distortion including an optical center of a lens using the photographed images, and to map each of the images to a surface of a Gaussian sphere using obtained optical center;

a virtual space arrangement unit configured to, using the relative location and installation angle of the camera estimated by the image arrangement estimation unit, change an angle and distance of the Gaussian sphere so that the Gaussian sphere coincides with the real reference patterns, and arrange the Gaussian spheres in a 3D virtual space; and

a single image acquisition unit configured to obtain a single 3D image by mapping the images arranged in the 3D virtual space onto an inner surface of a 3D sphere corresponding to a single large sphere.

2. The apparatus of claim 1, wherein the image arrangement estimation unit sets information about the estimated relative location and installation angle of the camera.

3. The apparatus of claim 1, wherein the optical parameters comprise one or more of an optical center, aspect ratio, an image sensor, a projection type and focal length.

4. The apparatus of claim 1, wherein the virtual space arrangement unit uses an optimization algorithm for estimating change parameters when changing an angle of rotation and distance of movement of the Gaussian sphere.

5. The apparatus of claim 4, wherein the optimization algorithm is any one of particle swarm optimization (PSO), gradient descent (GD), and least mean square estimation (LMSE).

6. The apparatus of claim 1, wherein the single image acquisition unit, in order to map an image corresponding to a central portion of each of the images, divides the surface of the 3D sphere by a number of cameras, and maps only an image from a most adjacent camera.

7. The apparatus of claim 1, wherein the single image acquisition unit divides the inner surface of the 3D sphere for individual longitudes based on a number of cameras, divides one surface of the 3D sphere divided for the individual longitudes into ground and non-ground planes, and performs mapping for individual latitudes.

8. The apparatus of claim 7, wherein mapping methods for ground and non-ground planes differ from each other in the 3D sphere.

9. A method of synthesizing 3D images, comprising:

receiving, by a plurality of wide angle cameras mounted on a vehicle, a plurality of images of reference patterns formed on a ground;

extracting feature points from the photographed reference patterns, and estimating a relative location and installation angle of each of the cameras using known physical location information of the extracted feature points;

obtaining optical parameters for correction of lens distortion including an optical center of a lens using the photographed images, and mapping each of the images onto a surface of a Gaussian sphere using the obtained optical center;

using the estimated relative location and installation angle of the camera, changing an angle and distance of the Gaussian sphere so that the Gaussian sphere coincides with the real reference patterns, and arranging the Gaussian spheres in a 3D virtual space; and

obtaining a single 3D image by mapping the images arranged in the 3D virtual space onto an inner surface of a 3D sphere corresponding to a single large sphere.

10. The method of claim 9, wherein arranging the Gaussian spheres in the 3D virtual space is performed by using any one of PSO, GD, and LMSE algorithms in order to estimate change parameters when changing the angle of rotation and distance of movement of the Gaussian sphere.

11. The method of claim 9, wherein arranging the Gaussian spheres in the 3D virtual space comprises performing matching with the real reference patterns using the following equation:

12. The method of claim 9, wherein obtaining the single 3D image comprises mapping the images arranged in the 3D virtual space onto areas of the inner surface of the 3D sphere most adjacent to a center of the 3D sphere.

13. The method of claim 9, wherein obtaining the single 3D image comprises:

dividing the inner surface of the 3D sphere for individual longitudes based on a number of cameras in order to map images corresponding to central portions of the images;

dividing the inner surface of the 3D sphere divided for the individual longitudes into ground and non-ground planes based on a latitude corresponding to the ground plane; and

dividing the images arranged in the 3D virtual space into ground and non-ground planes, and performing mapping onto corresponding areas of the inner surface of the 3D sphere.

14. The method of claim 13, wherein performing the mapping is performed by using different mapping methods for the ground and non-ground planes.

15. The method of claim 13, wherein performing the mapping comprises, when performing mapping onto the ground plane of the 3D sphere, bringing a point (x_p,y_p, z_p) of the 3D sphere from a point of a most adjacent Gaussian sphere and mapping the point (x_p,y_p,z_p) of the 3D sphere onto the 3D sphere, using a method of calculating T of the following equation and then obtaining x,y,z:

16. The method of claim 13, wherein performing the mapping comprises, when performing mapping onto the ground plane of the 3D sphere, bringing a point (x_p,y_p,z_p) of the 3D sphere from a point of a most adjacent Gaussian sphere and mapping the point (x_p,y_p,z_p) of the 3D sphere onto the 3D sphere, using a method of calculating T of the following equation and then obtaining x,y,z: