Generating three-dimensional façade models from images

• US 9,098,926 B2
• Filed: 01/22/2010
• Issued: 08/04/2015
• Est. Priority Date: 02/06/2009
• Status: Active Grant

• Current Assignee: Hong Kong University of Science and Technology
• Original Assignee: Hong Kong University of Science and Technology
• Inventors: Quan, Long, Xiao, Jianxiong, Fang, Tian, Zhao, Peng
• Primary Examiner(s): Liu, Zhengxi
• Application Number:
  US12/692,155
• Publication Number:
  US 20100201682A1
• Time in Patent Office: 2,020 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06V 20/39 Urban scenes

Abstract

The subject disclosure relates to generating three-dimensional façade models from images. In one aspect, a systematic decomposition schema is disclosed for a façade structured into a Direct Acyclic Graph and implemented as a top-down recursive subdivision and bottom-up merging. In a further aspect, optimization of façade depth operating in a façade surface and in the super-pixel level of a whole subdivision region is described.

Claims

1. A method that facilitates image-based faç

ade modeling, comprising;

determining, by a system comprising a processor, a composite texture image associated with a sequence of captured images, including obtaining camera positions associated with the sequence of captured images and obtaining three dimensional (3D) points associated with the sequence of captured images used to determine the composite texture image;

decomposing, by the system, a data representation of a faç

ade that is based at least in part on the composite texture image, the camera positions, and the 3D points, by recursively subdividing the data representation of the faç

ade into a set of subdivided patches;

bottom-up merging of at least a subset of the set of subdivided patches by merging neighboring subdivided patches based at least in part on a determination that a measure of support to separate neighboring subdivided patches based on architectural bilateral symmetry is not enough;

structuring, after the bottom-up merging, the data representation of the faç

ade into a directed acyclic graph of rectilinear patches; and

augmenting the data representation of the faç

ade with depth information based on the sequence of captured images and a subset of the 3D points by determining, by the system, and assigning an optimized depth to at least one of the subdivided patches.

2. The method of claim 1, wherein the subdividing includes subdividing the data representation of the faç

ade into the set of subdivided patches based at least in part on recovering lines of hidden structure in the data representation of the faç

ade.

3. The method of claim 1, wherein the determining the composite texture image comprises employing a photo consistency constraint based on patch similarity to detect at least one occluding object in the sequence of captured images.

4. The method of claim 1, wherein the obtaining the camera positions and the obtaining the 3D points comprises computing the camera positions and the 3D points as a set of dense 3D points from the sequence of captured images.

5. The method of claim 1, wherein the determining the optimized depth comprises determining the optimized depth based on the 3D points associated with the sequence of captured images used to determine the composite texture image, and wherein the assigning includes receiving a depth assignment instruction related to a subset of the subdivided patches.

6. The method of claim 1, wherein the assigning includes determining a maximum a posteriori (MAP) estimation of a Markov Random Field (MRF) graph and minimizing an image and viewpoint independent constraint for the faç

ade based on the sequence of captured images and the 3D points.

7. A method, comprising:

extracting, by a system comprising a processor, hidden structure from a reference texture image by recovering segmented lines of hidden structure through application of a Hough transform;

weighting, by the system, the segmented lines of hidden structure according to the correspondence of the segmented lines of hidden structure with an edge map for the reference texture image to partition the reference texture image;

recursively subdividing the reference texture image by the system into a set of subdivided regions based at least in part on the partitioned reference texture image;

bottom-up merging, by the system, at least a subset of the set of subdivided regions by merging neighboring subdivided regions based at least in part on a determination that a measure of support to separate neighboring subdivided regions based on architectural bilateral symmetry is not enough;

structuring, by the system, the partitioned reference texture image into a directed acyclic graph of rectilinear regions associated with the reference texture image; and

augmenting a data representation of a faç

ade with depth information based on the reference texture image and a subset of three dimensional (3D) points associated with the reference texture image by determining and assigning an optimized depth to at least one of the set of subdivided regions.

8. The method of claim 7, wherein the subdividing includes subdividing the reference texture image into the set of subdivided regions based at least in part receiving partition refinement instructions.

9. The method of claim 7, further comprising:

performing an edge detection on the reference texture image to create the edge map.

10. A method of faç

ade augmentation in a faç

ade modeling system, comprising;

receiving, by a system comprising a processor, data associated with a sequence of captured images, including associated three dimensional (3D) points related to the sequence of captured images, representing a faç

ade structured into a directed acyclic graph and having a set of subdivided regions of a texture image based on the sequence of captured images;

determining, by the system, and assigning an optimized depth based on the sequence of captured images and the associated 3D points to at least one subdivision region in the set of subdivided regions based at least in part on a determined maximum a posteriori (MAP) estimation of a Markov Random Field (MRF) graph and a minimized image and viewpoint independent constraint associated with the texture image, wherein the MRF graph is defined based on the at least one subdivision region represented by a single site.

11. The method of claim 10, wherein the assigning includes receiving a depth assignment instruction related to the at least one subdivision region.

12. The method of claim 11, wherein the assigning includes assigning the optimized depth based at least in part on the depth assignment instruction related to the at least one subdivision region.

13. An image-based faç

ade modeling system comprising;

a memory to store computer-executable components; and

a processor communicatively coupled to the memory that facilitates execution of the computer-executable components, the computer-executable components comprising;

a texture compositing component that determines a composite texture image associated with a sequence of captured images, obtains camera positions associated with the sequence of captured images, and obtains three dimensional (3D) points associated with the sequence of captured images used to determine the composite texture image;

a faç

ade decomposition component that decomposes a data representation of a faç

ade that is based at least in part on the composite texture image, the camera positions, and the 3D points by recursively subdividing the data representation of the faç

ade into a set of subdivided regions, bottom-up merges at least a subset of the set of subdivided regions by merging neighboring subdivided regions based at least in part on a determination that a measure of support to separate neighboring subdivided regions based on architectural bilateral symmetry is not enough, and structures, after the bottom-up merging, the data representation of the faç

ade into a directed acyclic graph of rectilinear patches; and

a faç

ade augmentation component that assigns an optimized depth based on the sequence of captured images and associated 3D points related to the sequence of captured images to at least one subdivision regions in the set of subdivided regions.

14. The system of claim 13, wherein the computer-executable components further comprise:

a modeling component that provides, based at least in part on camera location information, an approximate model associated with the composite texture image to the faç

ade modeling system.

15. The system of claim 13, wherein the computer-executable components further comprise:

an interface component that facilitates receiving at least one of subdivision refinement instructions, depth assignment instructions, or guiding instructions for generating the composite texture image.