METHOD AND APPARATUS FOR CONSTRUCTING THREE DIMENSIONAL MODEL OF OBJECT

• US 20170046868A1
• Filed: 07/11/2016
• Published: 02/16/2017
• Est. Priority Date: 08/14/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: CHERNOV, Vitaly Vladimirovich, SHAMSUAROV, Artem Gennadievich, MURATOV, Oleg Fanilevich, SLYNKO, Yury Vyacheslavovich, LYUBIMTSEVA, Maria Mikhailovna, BUCHA, Victor Valentinovich
• Granted Patent: US 10,360,718 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 17/205 Re-meshing
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method and an apparatus for constructing a three-dimensional (3D) model of an object are provided. The method includes capturing images of the object by scanning the object along a trajectory around the object, estimating positions of a scanner, which respectively correspond to the captured images, refining the estimated positions of the scanner based on at least two locations on the trajectory, estimating depth maps corresponding to the refined positions of the scanner, generating a surface mesh of the object by fusing the depth maps, and constructing and displaying the 3D model of the object by mapping textures onto the generated surface mesh.

Claims

1. A method of constructing a three-dimensional (3D) model of an object, the method comprising:

capturing images of the object by scanning the object along a trajectory around the object;

estimating positions of a scanner, which respectively correspond to the captured images;

refining the estimated positions of the scanner based on at least two locations on the trajectory;

estimating depth maps corresponding to the refined positions of the scanner;

generating a surface mesh of the object by fusing the estimated depth maps; and

constructing and displaying the 3D model of the object by mapping textures onto the generated surface mesh.

2. The method of claim 1,

wherein the trajectory is a closed loop or an opened loop, and

wherein the at least two locations on the trajectory comprise start and end points of the trajectory, which correspond to loop closure points for a loop closure of the trajectory.

3. The method of claim 1, wherein the capturing of the images further comprises capturing the images by scanning the object in 360°

by using a simultaneous localization and mapping (SLAM).

4. The method of claim 1, wherein the estimating of the positions of the scanner further comprises:

performing map initialization by using a first image from among the captured images;

extracting key points from the first image by using a features from accelerated segment test (FAST) algorithm performing corner detection; and

estimating the positions of the scanner, which respectively correspond to the captured images by using the extracted key points.

5. The method of claim 4, wherein the estimating of the positions of the scanner further comprises:

determining whether each of the captured images corresponds to a key frame; and

storing a captured image determined to be the key frame in a memory by matching the captured image determined to be the key frame to each of the estimated positions of the scanner.

6. The method of claim 5, wherein the refining of the estimated positions further comprises:

obtaining loop points on the trajectory with respect to the key frame;

checking whether the loop points are loop closure points; and

when the loop points are the loop closure points, refining the estimated positions based on a result of performing a loop closure by using the loop closure points.

7. The method of claim 1, wherein the refining of the estimated positions is performed to compensate for an effect of a drift generated when the object is scanned by using a simultaneous localization and mapping (SLAM).

8. The method of claim 1, wherein the estimating of the depth maps further comprises:

selecting a stereo pair comprising adjacent key frames; and

estimating the depth maps from the selected stereo pair by using a pyramidal approach in which a result of low resolution sampling is up-sampled via sampling of finer resolution.

9. The method of claim 1, wherein the generating of the surface mesh further comprises:

generating the surface mesh corresponding to a voxel-to-octree representation regarding the object by fusing the estimated depth maps by using 3D voxel truncated signed distance function (TSDF).

10. The method of claim 1, wherein the constructing and displaying of the 3D model further comprises:

checking whether each face of the surface mesh is visible or invisible at each of the refined positions;

texturing a visible face of the surface mesh to a projection corresponding to one of the captured images based on a result of the checking; and

constructing and displaying the 3D model by generating the texturized surface mesh.

11. An apparatus for configuring a three-dimensional (3D) model of an object, the apparatus comprising:

a scanner configured to capture images of the object by scanning the object along a trajectory around the object;

at least one processor configured to;

estimate positions of the scanner, which respectively correspond to the captured images,

refine the estimated positions of the scanner based on at least two locations on the trajectory,

estimate depth maps corresponding to the refined locations of the scanner,

generate a surface mesh of the object by fusing the estimated depth maps, and

construct the 3D model of the object by mapping textures onto the generated surface mesh; and

a display configured to display the 3D mode of the object.

12. The apparatus of claim 11,

wherein the trajectory is a closed loop or an opened loop, and

wherein the at least two locations on the trajectory comprise start and end points of the trajectory, which correspond to loop closure points for a loop closure of the trajectory.

13. The apparatus of claim 11, wherein the scanner is further configured to capture the images by scanning the object in 360°

by using a simultaneous localization and mapping (SLAM).

14. The apparatus of claim 11, wherein the at least one processor is further configured to:

perform map initialization by using a first image from among the captured images,

extract key points from the first image by using a features from accelerated segment test (FAST) algorithm performing corner detection, and

estimate the positions of the scanner, which respectively correspond to the captured images by using the extracted key points.

15. The apparatus of claim 14, wherein the at least one processor is further configured to:

determine whether each of the captured images corresponds to a key frame, and

store a captured image determined to be the key frame in a memory by matching the captured image determined to be the key frame to each of the estimated positions of the scanner.

16. The apparatus of claim 15, wherein the at least one processor is further configured to:

obtain loop points on the trajectory with respect to the key frame,

check whether the loop points are loop closure points, and

when the loop points are the loop closure points, refine the estimated positions based on a result of performing loop closure by using the loop closure points.

17. The apparatus of claim 11, wherein the at least one processor is further configured to refine the positions of the scanner to compensate for an effect of a drift generated when the object is scanned by using a simultaneous localization and mapping (SLAM).

18. The apparatus of claim 11, wherein the at least one processor is further configured to:

select a stereo pair comprising adjacent key frames, and

estimate the depth maps from the selected stereo pair by using a pyramidal approach in which a result of low resolution sampling is up-sampled via sampling of finer resolution.

19. The apparatus of claim 11, wherein the at least one processor is further configured to generate the surface mesh corresponding to a voxel-to-octree representation regarding the object by fusing the estimated depth maps by using 3D voxel truncated signed distance function (TSDF).

20. The apparatus of claim 11, wherein the at least one processor is further configured to:

check whether each face of the surface mesh is visible or invisible at each of the refined positions,

texture a visible face of the surface mesh to a projection corresponding to one of the captured images based on a result of the checking, and

construct the 3D model by generating the texturized surface mesh.