GENERATING THREE-DIMENSIONAL MODELS FROM IMAGES

• US 20120041722A1
• Filed: 02/03/2010
• Published: 02/16/2012
• Est. Priority Date: 02/06/2009
• Status: Active Grant

• Current Assignee: Hong Kong University of Science and Technology
• Original Assignee: Hong Kong University of Science and Technology
• Inventors: Quan, Long, Xiao, Jianxiong, Fang, Tian, Zhao, Peng
• Granted Patent: US 8,798,965 B2
• Time in Patent Office: Days
• US Class Current: 703/1
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/579 from motion

Abstract

The subject disclosure relates to generating models from images. In an aspect, multi-view semantic segmentation is provided to recognize and segment images at the pixel level into semantically meaningful areas, and which can provide labels with a specific object class. In further aspects, a partition scheme is provided that can separate objects into independent blocks using major line structures of a scene. In addition, an inverse patch-based orthographic composition and structure analysis on a block is provided that can regularize noisy and missing reconstructed 3D data to facilitate image-based modeling.

Claims

1. A method that facilitates image-based modeling, comprising:

receiving input image data representing a faç

ade;

reconstructing input image data with a computing device to compute three-dimensional (3D) points, lines, and camera positions associated with the faç

ade; and

performing a multi-view semantic segmentation on reconstructed input image data to recognize faç

ade structure and segment the faç

ade.

2. The method of claim 1, further comprising:

receiving a segmentation instruction to interactively refine the multi-view semantic segmentation.

3. The method of claim 1, further comprising:

block partitioning reconstructed input image data to produce at least one individual building block associated with the segmented faç

ade.

4. The method of claim 3, further comprising:

performing an inverse orthographic composition on reconstructed input image data associated with the at least one individual building block to produce composed orthographic depth map and texture for the at least one individual building block.

5. The method of claim 4, further comprising:

receiving an inpainting instruction to interactively edit at least one of the composed orthographic depth map or texture.

6. The method of claim 4, further comprising:

performing structural analysis and regularization of the composed orthographic depth map and texture to identify structural elements at different faç

ade depths for the at least one individual building block.

7. The method of claim 6, further comprising:

generating a 3d model by generating geometry for the at least one individual building block from the identified structural elements at different faç

ade depths and texturing the at least one individual building block.

8. The method of claim 7, further comprising:

generating a city model by combining the generated 3d model for the at least one individual building block with a 3d model for at least one other individual building block.

9. An image-based modeling system comprising:

a multi-view semantic segmentation component configured to create a semantic segmentation of a faç

ade based at least in part on a registered sequence of images associated with the faç

ade.

10. The system of claim 9, further comprising:

a partitioning component configured to block partitions reconstructed image data to determine at least one object block associated with the semantic segmentation of the faç

ade.

11. The system of claim 9, further comprising:

an image reconstruction component configured to register a sequence of images representing the faç

ade to produce the registered sequence of images and that compute the reconstructed image data comprising three-dimensional (3D) points, lines, and camera positions associated with the faç

ade.

12. The system of claim 9, further comprising:

an inverse orthographic composition component configured to compose an orthographic depth map and texture from the reconstructed image data for the at least one object block.

13. The system of claim 12, further comprising:

a structural analysis and regularization component configured to determine structural elements at various faç

ade depths from the orthographic depth map and texture for the at least one object block.

14. The system of claim 13, further comprising:

a modeling component configured to generate block geometry for the at least one object block from the determined structural elements at various faç

ade depths.

15. The system of claim 14, the modeling component is further configured to texture the at least one object block to create an object block model associated with the faç

ade.

16. The system of claim 14, the modeling component is further configured to merge the object block model associated with the faç

ade with at least one other object block model associated with the faç

ade to compose a composite faç

ade model.

17. The system of claim 11, further comprising:

an interface component configured to receive the sequence of images representing the faç

ade.

18. The system of claim 17, the interface component is further configured to receive instructions that, in response to execution by the image-based modeling system, at least one of interactively refine the multi-view semantic segmentation, or interactively edit at least one of a depth map or a texture associated with the faç

ade.

19. An image-based modeling method, comprising:

performing a multi-view semantic segmentation on at least a subset of reconstructed input image data representing a faç

ade to recognize faç

ade structure and segment the faç

ade;

block partitioning the at least a subset of the reconstructed input image data to produce at least one object block associated with the segmented faç

ade;

performing an inverse orthographic composition on the at least a subset of the reconstructed input image data associated with the at least one object block to produce a composed orthographic depth map and a composed orthographic texture for the at least one object block; and

performing structural analysis and regularization of the composed orthographic depth map and the composed orthographic texture to identify structural elements at a plurality of faç

ade depths for the at least one object block.

20. The method of claim 19, further comprising:

generating an object model by generating geometry for the at least one object block from the identified structural elements at the plurality of faç

ade depths and texturing the at least one object block.