POINT CLOUD NOISE AND OUTLIER REMOVAL FOR IMAGE-BASED 3D RECONSTRUCTION

• US 20180096463A1
• Filed: 09/30/2016
• Published: 04/05/2018
• Est. Priority Date: 09/30/2016
• Status: Active Grant

• Current Assignee: ETH Zurich, Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: ETH Zurich, Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Kim, Changil, Sorkine-Hornung, Olga, Schroers, Christopher, Zimmer, Henning, Wolff, Katja, Botsch, Mario, Sorkine-Hornung, Alexander
• Granted Patent: US 10,074,160 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 15/06 Ray-tracing
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: G06T 5/70 Denoising; Smoothing

Abstract

Enhanced removing of noise and outliers from one or more point sets generated by image-based 3D reconstruction techniques is provided. In accordance with the disclosure, input images and corresponding depth maps can be used to remove pixels that are geometrically and/or photometrically inconsistent with the colored surface implied by the input images. This allows standard surface reconstruction methods (such as Poisson surface reconstruction) to perform less smoothing and thus achieve higher quality surfaces with more features. In some implementations, the enhanced point-cloud noise removal in accordance with the disclosure can include computing per-view depth maps, and detecting and removing noisy points and outliers from each per-view point cloud by checking if points are consistent with the surface implied by the other input views.

Claims

1. A method for reconstructing a three-dimensional (3D) model of an object, the method being performed by a computer system and comprising:

receiving images of the object, the images capturing the object from different viewpoints;

obtaining a depth map for each of the images, the depth map including information relating to a surface of the object in the 3D space;

projecting each pixel in each depth map into a 3D space to obtain a point set;

for each point in the point set, determining whether the point is consistent with the surface of the object in the 3D space as indicated by a set of depth maps for the images, wherein the set of depth maps does not include the depth map from which the point is projected, wherein the consistency determination includes;

for each depth map in the set;

performing an intersection of a ray with the surface of the object indicated by the depth map to find an intersection point, the ray being through a camera viewpoint corresponding to the depth map, and the point; and

in response to the intersection point being found, determining a distance between the point and intersection point as a distance between the point and the surface of the object as seen from the camera viewpoint corresponding to the depth map;

determining whether the point is geometrically consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object; and

determining whether to discard the point from point set based on the determination whether the point is geometrically consistent with the surface of the object; and

reconstructing the 3D model using the point set.

2. The method of claim 1, wherein the determination whether the point is geometrically consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object includes:

averaging distances between the point and the surface of the object as indicated by the set of depth maps to obtain an average distance between the point and the surface of the object.

3. The method of claim 2, wherein the determination whether the point is geometrically consistent with the surface of the object is further based on a threshold distance value such that the point is determined to be geometrically consistent when the average distance between the point and the surface of the object is less than or equal to the threshold distance value.

4. The method of claim 1, wherein whether the point is geometrically consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object includes:

truncating any distance between the point and the surface as indicated by a given depth map in the set to a preset value when the distance exceeds a threshold value.

5. The method of claim 1, wherein determining whether the point is geometrically consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object includes:

assigning a weight to each of the determined distances, and determining the distance between the point and the surface of the object based on the weighted distances.

6. The method of claim 1, wherein determining whether the point is geometrically consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object includes:

determining whether any distance between the point and the surface as indicated by a given depth map in the set is smaller than a threshold value indicating a negative distance value, and in response to determining the distance is smaller than the threshold value, excluding such a distance from the determination whether the point is geometrically consistent with the surface of the object.

7. The method of claim 1, further comprising:

for each point in the point set, determining whether the point is photometrically consistent with the surface of the object based on the intersection points found; and

, wherein

the determination whether to discard the point from the point set is further based on whether the point is photometrically consistent the surface of the object.

8. The method of claim 7, wherein determining whether the point is photometrically consistent with the surface of the object based on the intersection points found includes:

for each intersection point found;

determining whether the distance between the point and intersection point is larger than a threshold value; and

in response to determining the distance between the point and intersection point is larger than the threshold value, excluding the intersection point from the determination whether the point is photometrically consistent with the surface of the object.

9. The method of claim 7, further comprising:

for each point in the point set, determining whether the point is visible with respect to each of the depth maps in the set; and

, wherein

the determination whether the point is photometrically consistent with the surface of the object is further based on whether the point is visible with respect to each of the depth maps in the set.

10. The method of claim 7, wherein the determination whether the point is photometrically consistent with the surface of the object based on the intersection points found is based on a standard deviation between the color values of the point and the intersection points.

12. The system of claim 10, wherein the determination whether the point is geometrically consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object includes:

averaging distances between the point and the surface of the object as indicated by the set of depth maps to obtain an average distance between the point and the surface of the object.

11. A system for reconstructing a three-dimensional (3D) model of an object, the system comprising one or more of a processor configured to execute machine-readable instructions such that when the machine-readable instructions are executed by the processor, the processor is caused to perform:

receiving images of the object, the images capturing the object from different viewpoints;

obtaining a depth map for each of the images, the depth map including information relating to a surface of the object in the 3D space;

projecting each pixel point in each depth map into a 3D space to obtain a point set;

for each point in the point set, determining whether the point is consistent with the surface of the object in the 3D space as indicated by a set of depth maps for the images, wherein the set of depth maps does not include the depth map from which the point is projected, wherein the consistency determination includes;

for each depth map in the set;

performing an intersection of a ray with the surface of the object indicated by the depth map to find an intersection point, the ray being through a camera viewpoint corresponding to the depth map, and the point; and

in response to the intersection point being found, determining a distance between the point and intersection point as a distance between the point and the surface of the object as seen from the camera viewpoint corresponding to the depth map;

determining whether the point is geometrically consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object; and

determining whether to discard the point from point set based on the determination whether the point is geometrically consistent with the surface of the object; and

reconstructing the 3D model using the point set.

13. The system of claim 11, wherein the determination whether the point is geometrically consistent with the surface of the object is further based on a threshold distance value such that the point is determined to be geometrically consistent when the average distance between the point and the surface of the object is less than or equal to the threshold distance value.

14. The system of claim 11, wherein whether the point is geometrically consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object includes:

truncating any distance between the point and the surface as indicated by a given depth map in the set to a preset value when the distance exceeds a threshold value.

15. The system of claim 11, wherein determining whether the point is geometrically consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object includes:

assigning a weight to each of the determined distances, and determining the distance between the point and the surface of the object based on the weighted distances.

16. The system of claim 11, wherein determining whether the point is geometrically consistent with the surface of the object based on one or more of the determined distances between the point and the surface of the object includes:

determining whether any distance between the point and the surface as indicated by a given depth map in the set is smaller than a threshold value indicating a negative distance value, and in response to determining the distance is smaller than the threshold value, excluding such a distance from the determination whether the point is geometrically consistent with the surface of the object.

17. The system of claim 11, wherein when executing the machine-readable instructions, the processor is further caused to perform:

for each point in the point set, determining whether the point is photometrically consistent with the surface of the object based on the intersection points found; and

, wherein

the determination whether to discard the point from the point set is further based on whether the point is photometrically consistent with the surface of the object.

18. The system of claim 17, wherein determining whether the point is photometrically consistent with the surface of the object based on the intersection points found includes:

for each intersection point found;

determining whether the distance between the point and intersection point is larger than a threshold value; and

in response to determining the distance between the point and intersection point is larger than the threshold value, excluding the intersection point from the determination whether the point is photometrically consistent with the surface of the object.

19. The system of claim 17, wherein when executing the machine-readable instructions, the processor is further caused to perform:

for each point in the point set, determining whether the point is visible with respect to each of the depth maps in the set; and

, wherein

the determination whether the point is photometrically consistent with the surface of the object is further based on whether the point is visible with respect to each of the depth maps in the set.

20. The system of claim 17, wherein the determination whether the point is photometrically consistent with the surface of the object based on the intersection points found is based on a standard deviation between the color values of the point and the intersection points.