Modeling method and apparatus using three-dimensional (3D) point cloud

• US 10,217,234 B2
• Filed: 06/30/2017
• Issued: 02/26/2019
• Est. Priority Date: 01/26/2017
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Lee, Kyoobin, Kim, Seung-Chan, Hong, Sunghoon
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US15/638,518
• Publication Number:
  US 20180211399A1
• Time in Patent Office: 606 Days
• CPC Class Codes: G06F 30/00 Computer-aided design [CAD]
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/529 from texture
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

A modeling method using a three-dimensional (3D) point cloud, the modeling method including extracting at least one region from an image captured by a camera; receiving pose information of the camera based on two-dimensional (2D) feature points extracted from the image; estimating first depth information of the image based on the at least one region and the pose information of the camera; and generating a 3D point cloud to model a map corresponding to the image based on the first depth information.

Claims

1. A modeling method using a three-dimensional (3D) point cloud, the modeling method comprising:

extracting at least one region from an image captured by a camera;

receiving pose information of the camera that is tracked based on two-dimensional (2D) feature points extracted from the image;

estimating first depth information of the image based on the at least one region and the pose information of the camera; and

generating a 3D point cloud to model a map corresponding to the image based on the first depth information.

2. The modeling method of claim 1, wherein the pose information of the camera comprises either one or both of a key frame corresponding to the image and a pose variation that changes in response to a movement of the camera from a first position to a second position.

3. The modeling method of claim 2, wherein the estimating comprises:

determining, based on the pose information of the camera, whether to, either one or both of, replace the key frame with a new key frame and redefine the key frame in response to the movement of the camera from the first position to the second position; and

estimating either one or both of depth information related to the new key frame and depth information related to the redefined key frame to be the first depth information of the image based on a result of the determining.

4. The modeling method of claim 3, wherein the determining comprises:

replacing a key frame with respect to the at least one region with the new key frame in response to the pose information of the camera that changes in response to the movement of the camera being greater than or equal to a preset pose variation; and

redefining the key frame, with respect to the at least one region, in response to the pose information of the camera that changes in response to the movement of the camera being less than the preset pose variation.

5. The modeling method of claim 4, wherein the replacing comprises replacing depth information related to the new key frame by projecting points, corresponding to the key frame, with respect to the at least one region on the new key frame.

6. The modeling method of claim 4, wherein the redefining comprises redefining depth information related to the key frame by filtering pixels of the key frame combined with an interleaved spatial regularization through a baseline stereo comparison.

7. The modeling method of claim 1, further comprising:

fusing the first depth information and second depth information estimated based on the 2D feature points,

wherein the generating comprises generating the 3D point cloud to modeling the map corresponding to the image based on the fused depth information.

8. The modeling method of claim 7, wherein the fusing comprises fusing the first depth information and the second depth information by adding the second depth information to the first depth information.

9. The modeling method of claim 1, wherein the extracting comprises searching the image for at least one region including a portion having an intensity gradient variation corresponding to a first-order differential of a pixel intensity exceeding a preset reference value.

10. The modeling method of claim 1, further comprising:

reconstructing a 3D surface of the image based on the 3D point cloud.

11. The modeling method of claim 1, further comprising:

extracting parameters to model a target object from the 3D point cloud; and

modeling the target object based on the parameters.

12. A non-transitory computer-readable medium storing program instructions for controlling a processor to perform the method of claim 1.

15. The modeling method of claim 12, wherein the generating of the 3D point comprises:

projecting 3D points of a 3D curve, predicted through the 2D contour on a 2D image, based on the pose information of the camera, an intrinsic parameter of the camera, and the 2D feature points of which the 3D coordinates are verified;

calculating a coefficient of a curve parameter of the 3D curve that minimizes an error of an error function with respect to the 3D points projected on the 2D image, the error of the error function being zero when the 3D points projected on the 2D image match the 2D contour; and

performing curve fitting with respect to the 3D space of the 2D feature points based on the coefficient of the curve parameter of the 3D curve.

16. The modeling method of claim 15, wherein the error function is based on at least one of a 3D point corresponding to any one or any combination of any two or more of a frame of the image, the intrinsic parameter of the camera, and the pose information of the camera.

13. A modeling method using a three-dimensional (3D) point cloud, the modeling method comprising:

detecting an edge from an image captured by a camera;

receiving information related to two-dimensional (2D) feature points extracted from the image;

analyzing a 2D contour included in the image based on the edge and the information related to the 2D feature points;

receiving pose information of the camera that is tracked based on the 2D feature points;

generating a 3D point with respect to at least a portion of the 2D contour by performing curve fitting with respect to a 3D space corresponding to the 2D contour based on a result of the analyzing and the pose information of the camera; and

generating a 3D point cloud to model a map corresponding to the image by fusing a 3D point estimated based on the 2D feature points and the 3D point generated by curve fitting.

14. The modeling method of claim 13, wherein the analyzing comprises:

analyzing whether 2D feature points, of which 3D coordinates are verified through depth estimation, are positioned on or adjacent to a 2D contour of the edge by matching coordinates of the 2D feature points to the edge; and

determining 3D coordinates of 2D feature points positioned on, or adjacent to the 2D contour, based on the 2D feature points of which the 3D coordinates are verified.

17. The modeling method of claim 13, wherein the analyzing comprises:

obtaining a set of 2D curvatures corresponding to the 2D contour by analyzing the edge;

determining whether a curvature of the edge exceeds a preset curvature and a length of the edge exceeds a preset contour length; and

generating a contour library including any one or any combination of any two or more of coordinates of 2D feature points matched to the edge, the length of the edge, and the curvature of the edge based on a result of the determining.

18. The modeling method of claim 13, further comprising:

reconstructing a 3D surface of the image based on the 3D point cloud.

19. The modeling method of claim 13, further comprising:

extracting parameters to model a target object from the 3D point cloud; and

modeling the target object based on the parameters.

20. A modeling apparatus using a three-dimensional (3D) point cloud, the modeling apparatus comprising:

a communication interface configured to receive pose information of a camera that is tracked based on two-dimensional (2D) feature points extracted from an image captured by the camera; and

a processor configured;

to extract at least one region from the image,

to estimate first depth information of the image based on the at least one region and the pose information of the camera, and

to generate a 3D point cloud for modeling a map corresponding to the image based on the first depth information.

21. A modeling method using a three-dimensional (3D) point cloud, the modeling method comprising:

generating a first depth information, the first depth information being generated based on an identified edge of at least a region of an image captured by a camera;

receiving an estimated second depth information, based on an identified two-dimensional (2D) point of the region of the image captured by the camera;

fusing the second depth information with the first depth information; and

,

generating a 3D point cloud to model a map of the region of the image based on the fused first and second depth information.

22. The modeling method of claim 21, wherein the second depth information is generated based on a comparison of pose information of the camera and positions of identified 2D feature points amongst a plurality of images captured by the camera.

23. The modeling method of claim 21, wherein the first depth information is generated based on a detected edge within the image, the edge being analyzed as a 2D contour transformatively applied as a 3D curve through successive approximations.

24. The modeling method of claim 21, further comprising:

transformatively projecting modified imagery comprising reconstructed 3D surfaces of the image based on the map of the region of the image to generate either one or both of a virtual reality and an augmented reality view for a user.