Generating three-dimensional models from images

• US 8,798,965 B2
• Filed: 02/03/2010
• Issued: 08/05/2014
• Est. Priority Date: 02/06/2009
• Status: Active Grant

• Current Assignee: Hong Kong University of Science and Technology
• Original Assignee: Hong Kong University of Science and Technology
• Inventors: Quan, Long, Xiao, Jianxiong, Fang, Tian, Zhao, Peng
• Primary Examiner(s): Shah, Kamini S
• Assistant Examiner(s): PIERRE LOUIS, ANDRE
• Application Number:
  US13/148,173
• Publication Number:
  US 20120041722A1
• Time in Patent Office: 1,644 Days
• Field of Search: 703/1, 703/2, 382/154, 382/171, 382/173, 382/254, 382/260, 345/419, 345/420, 345/427
• US Class Current: 703/1
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/579 from motion

Abstract

The subject disclosure relates to generating models from images. In an aspect, multi-view semantic segmentation is provided to recognize and segment images at the pixel level into semantically meaningful areas, and which can provide labels with a specific object class. In further aspects, a partition scheme is provided that can separate objects into independent blocks using major line structures of a scene. In addition, an inverse patch-based orthographic composition and structure analysis on a block is provided that can regularize noisy and missing reconstructed 3D data to facilitate image-based modeling.

Claims

1. A method, comprising:

receiving, by a computing device comprising a processor, input image data, wherein the input image data represents a faç

ade;

reconstructing, by the computing device, the input image data comprising determining three-dimensional (3D) points, lines, and camera positions associated with the faç

ade;

performing, by the computing device, a multi-view semantic segmentation on the reconstructed input image data comprising;

recognizing a faç

ade structure; and

segmenting the faç

ade as a result of the recognizing;

block partitioning, by the computing device, the reconstructed input image data into a plurality of independent object blocks comprising a first object block and at least a second object block, comprising ignoring vertical line segments comprising extensions that cross a defined number of horizontal line segments; and

separating, by the computing device, the first object block from at least the second object block comprising using one or more line structures in the input image data, wherein the first object block and at least the second object block belong to a same object class.

2. The method of claim 1, further comprising:

receiving, by the computing device, a segmentation instruction; and

refining, by the computing device, the multi-view semantic segmentation in response to the segmentation instruction being received.

3. The method of claim 1, further comprising:

performing, by the computing device, an inverse orthographic composition on the reconstructed input image data associated with the first object block and at least the second object block; and

producing, by the computing device, a composed orthographic depth map and texture for the first object block and at least the second object block based on the inverse orthographic composition.

4. The method of claim 3, further comprising:

receiving, by the computing device, an inpainting instruction; and

editing, by the computing device, at least one of the composed orthographic depth map or texture based on the inpainting instruction.

5. The method of claim 3, further comprising:

performing, by the computing device, structural analysis and regularization of the composed orthographic depth map and texture; and

identifying, by the computing device, one or more structural elements at different faç

ade depths for the first object block and at least the second object block.

6. The method of claim 5, further comprising:

generating, by the computing device, a 3D model comprising generating geometry for the first object block and at least the second object block from the identified one or more structural elements at the different faç

ade depths and texturing the first object block and at least the second object block.

7. The method of claim 6, further comprising:

generating, by the computing device, a city model comprising combining the generated 3D model for the first object block and at least the second object block with a 3D model for at least a third object block.

8. A system comprising:

a processor that executes or facilitates execution of computer executable components stored in a memory, the computer executable components comprising;

a multi-view semantic segmentation component configured to create a semantic segmentation of a faç

ade based at least in part on a registered sequence of images associated with the faç

ade;

a partitioning component configured to block partition the semantic segmentation into a plurality of object blocks comprising;

determining a number of times that a first vertical line crosses horizontal lines within the faç

ade, and

determining the number of times satisfies a threshold amount, wherein the partitioning component ignores the first vertical line for the block partition; and

a block separator component configured to distinguish a first object block of the plurality of object blocks from a second object block of the plurality of object blocks, wherein the first object block and the second object block are classified in a same object class.

9. The system of claim 8, further comprising:

an image reconstruction component configured to register a sequence of images representing the faç

ade and produce the registered sequence of images, wherein the image reconstruction component is also configured to compute reconstructed image data comprising three-dimensional (3D) points, lines, and camera positions associated with the faç

ade.

10. The system of claim 8, further comprising:

an inverse orthographic composition component configured to compose an orthographic depth map and texture from reconstructed image data for the first object block.

11. The system of claim 10, further comprising:

a structural analysis and regularization component configured to determine structural elements at two or more different faç

ade depths from the orthographic depth map and the texture for the first object block.

12. The system of claim 11, further comprising:

a modeling component configured to generate block geometry for the first object block from the determined structural elements at the two or more different faç

ade depths.

13. The system of claim 12, the modeling component is further configured to texture the first object block and create an object block model associated with the faç

ade.

14. The system of claim 12, the modeling component is further configured to merge an object block model associated with the faç

ade with at least one other object block model associated with the faç

ade and compose a composite faç

ade model.

15. The system of claim 9, further comprising:

an interface component configured to receive the sequence of images representing the faç

ade.

16. The system of claim 15, the interface component is further configured to receive instructions that, in response to execution by the system, at least one of refine the semantic segmentation of the faç

ade, or edit at least one of a depth map or a texture associated with the faç

ade.

17. A method, comprising:

performing, by a system comprising a processor, a multi-view semantic segmentation on at least a subset of reconstructed input image data representing a faç

ade, wherein the performing comprises recognizing faç

ade structure and segmenting the faç

ade based on the recognizing resulting in a segmented faç

ade;

block partitioning, by the system, at least the subset of reconstructed input image data comprising;

determining respective scores for a set of vertical lines in the reconstructed input image data, wherein the respective scores represent a number of times each vertical line of the set of vertical lines crosses horizontal lines in the reconstructed input image data,

selecting a subset of vertical lines from the set of vertical lines based in part on the respective scores, the selecting comprising ignoring vertical line segments comprises extensions that cross a defined number of horizontal line segments, and

producing at least one object block associated with the segmented faç

ade based in part on the subset of vertical lines;

performing, by the system, an inverse orthographic composition on at least the subset of reconstructed input image data associated with the at least one object block comprising producing a composed orthographic depth map and a composed orthographic texture for the at least one object block; and

performing, by the system, structural analysis and regularization of the composed orthographic depth map and the composed orthographic texture comprising identifying structural elements at a plurality of faç

ade depths for the at least one object block.

18. The method of claim 17, further comprising:

generating, by the system, an object model comprising generating geometry for the at least one object block from the identified structural elements at the plurality of faç

ade depths and texturing the at least one object block.

19. The method of claim 17, wherein the block partitioning further comprising:

producing at least another object block associated with the segmented faç

ade, wherein the at least one object block and the at least another object block are adjacent in the segmented faç

ade and belong to a same object class; and

separating the at least one object block from the at least another object block.

20. The method of claim 19, wherein the separating comprises using line structures in the segmented faç

ade to partition the at least one object block from the at least another object block.