STRUCTURE DEPTH-AWARE WEIGHTING IN BUNDLE ADJUSTMENT

• US 20200134847A1
• Filed: 01/03/2019
• Published: 04/30/2020
• Est. Priority Date: 10/31/2018
• Status: Active Grant

• Current Assignee: Bentley Systems Incorporated
• Original Assignee: Bentley Systems Incorporated
• Inventors: Gros, Nicolas
• Granted Patent: US 10,957,062 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

In various embodiments, techniques are provided for photogrammetric 3D model reconstruction that modify the optimization performed in bundle adjustment operations of an automatic SfM stage to apply a depth-aware weighting to reprojection error of each 3D point used in the optimization. The reprojection error of each 3D point may be weighted based on a function of distance, density of a cluster, or a combination of distance and density. A loss function may be scaled to account for the weighting, and normalizations applied. Such weighting may force consideration of 3D points on an object of interest in the foreground and improve convergence of the optimization to global optima. In such manner, accurate and complete 3D models may be reconstructed of even ill-textured or very thin objects in the foreground of a scene with a highly textured background, while not consuming excessive processing and storage resources or requiring tedious workflows.

Claims

1. A method for performing photogrammetric 3D model reconstruction, comprising obtaining, by 3D model reconstruction software executing on an electronic device, a set images of a scene that include an object of interest taken by a camera from different viewpoints;

automatically selecting keypoints in the sets of images;

automatically matching corresponding keypoints that appear in more than one of the set of images;

estimating 3D points in 3D space for features of the scene represented by corresponding keypoints in the images;

performing bundle adjustment operations to simultaneously refine the estimated 3D points and camera parameters for the set of images, the bundle adjustment operations performed as an optimization with a loss function that penalizes reprojection error, wherein a depth-aware weighting is applied to the reprojection error of each 3D point in the optimization;

utilizing the refined estimated 3D positions and camera parameters produced by the bundle adjustment operations in a dense 3D reconstruction to produce a 3D model that includes the object; and

displaying the 3D model, by the 3D model reconstruction software executing on the electronic device, on a display screen.

2. The method of claim 1, wherein the depth-aware weighting comprises a weighting based on a function of distance between the respective 3D point and the camera.

3. The method of claim 2, wherein the function of distance is an inverse of distance between the respective 3D point and the camera.

4. The method of claim 1, wherein performing the bundle adjustment operations comprise applying a clustering algorithm to partition the 3D points into a number of clusters based on their distance to the camera, and the depth-aware weighting comprises a weighting based on a function of density of the cluster of the respective 3D point.

5. The method of claim 4, wherein the function of density is an inverse of a number of points of the cluster of the respective 3D point.

6. The method of claim 4, wherein the clustering algorithm is a one-dimensional (1D) k-mean clustering algorithm.

7. The method of claim 1, wherein performing the bundle adjustment operations comprises applying a clustering algorithm to partition the 3D points into a number of clusters based on their distance to the camera, and the depth-aware weighting comprises a weighting based on a function that combines distance between the respective 3D point and the camera and density of the cluster of the respective 3D point.

8. The method of claim 1, wherein the object is an object in the foreground of the scene, and the depth-aware weighting is applied to the reprojection error to increase a number of 3D points retained by bundle adjustment operations on the object in the foreground.

9. The method of claim 1, wherein object is piece of infrastructure.

10. The method of claim 1, wherein the camera is part of an aerial drone, and the obtaining comprises:

receiving the set images from the aerial drone.

11. The method of claim 1, wherein the 3D model reconstruction software is a client application, and the automatically selecting keypoints, automatically matching corresponding keypoints, estimating 3D points, and performing the bundle adjustment operations are performed by backend 3D model reconstruction processing software executing on a server or cloud services architecture accessible to the electronic device over a network.

12. The method of claim 1, wherein the 3D model is a textured 3D mesh.

13. A system for performing photogrammetric three-dimensional (3D) model reconstruction, comprising:

a camera configured to capture a set images of a scene that include an object of interest taken from different viewpoints; and

one or more electronic devices configured to execute 3D model reconstruction software, the 3D model reconstruction software when executed operable to;

automatically select keypoints in the sets of images,

automatically match corresponding keypoints that appear in more than one of the set of images,

estimate 3D points in 3D space for features of the scene represented by corresponding keypoints in the images,

perform bundle adjustment operations to simultaneously refine the estimated 3D points and camera parameters for the set of images, the bundle adjustment operations performed as an optimization with a loss function that penalizes reprojection error, wherein a depth-aware weighting is applied to the reprojection error of each 3D point in the optimization, and

utilize the refined estimated 3D positions and camera parameters produced by the bundle adjustment operations in a dense 3D reconstruction to produce a 3D model that includes the object; and

a display device of the one or more electronic devices operable to display the 3D model.

14. The system of claim 13, wherein the depth-aware weighting comprises a weighting based on a function of distance between the respective 3D point and the camera.

15. The system of claim 13, wherein the 3D model reconstruction software when executed is further operable to apply a clustering algorithm to partition the 3D points into a number of clusters based on their distance to the camera, and the depth-aware weighting comprises a weighting based on a function of density of the cluster of the respective 3D point.

16. The system of claim 13, wherein the 3D model reconstruction software when executed is further operable to apply a clustering algorithm to partition the 3D points into a number of clusters based on their distance to the camera, and the depth-aware weighting comprises a weighting based on a function that combines distance between the respective 3D point and the camera and density of the cluster of the respective 3D point.

17. A non-transitory electronic device readable medium storing software for execution on one or more processors or one or more electronic devices, the software when executed operable to:

obtain a set images of a scene that include an object of interest taken by a camera from different viewpoints;

photogrammetrically reconstruct a three-dimensional (3D) model from the set of images utilizing an automatic structure-from-motion (SfM) stage and a dense 3D reconstruction stage, the automatic SfM stage including bundle adjustment operations that simultaneously refine estimated 3D points and camera parameters for the set of images by performing an optimization with a loss function that penalizes reprojection error, wherein a depth-aware weighting is applied to the reprojection error of each 3D point in the optimization, the dense 3D reconstruction stage to utilize the refined estimated 3D positions and camera parameters produced by the bundle adjustment operations to produce a 3D model that includes the object; and

is output the 3D model.

18. The non-transitory electronic device readable medium of claim 17, wherein the depth-aware weighting comprises a weighting based on a function of distance between the respective 3D point and the camera.

19. The non-transitory electronic device readable medium of claim 17, wherein the software when executed is further operable to apply a clustering algorithm to partition the 3D points into a number of clusters based on their distance to the camera, and the depth-aware weighting comprises a weighting based on a function of density of the cluster of the respective 3D point.

20. The non-transitory electronic device readable medium of claim 17, wherein the 3D model reconstruction software when executed is further operable to apply a clustering algorithm to partition the 3D points into a number of clusters based on their distance to the camera, and the depth-aware weighting comprises a weighting based on a function that combines distance between the respective 3D point and the camera and density of the cluster of the respective 3D point.