Using photo collections for three dimensional modeling

• US 8,350,850 B2
• Filed: 03/31/2008
• Issued: 01/08/2013
• Est. Priority Date: 03/31/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Steedly, Drew, Szeliski, Richard, Sinha, Sudipta, Agrawala, Maneesh
• Primary Examiner(s): HAJNIK, DANIEL F
• Application Number:
  US12/059,794
• Publication Number:
  US 20090244062A1
• Time in Patent Office: 1,744 Days
• Field of Search: None
• US Class Current: 345/420
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06T 7/564 from contours

Abstract

A collection of photos and a three-dimensional reconstruction of the photos are used to construct and texture a mesh model. In one embodiment, a first digital image of a first view of a real world scene is analyzed to identify lines in the first view. Among the lines, parallel lines are identified. A three-dimensional vanishing direction in a three-dimensional space is determined based on the parallel lines and an orientation of the digital image in the three-dimensional space. A plane is automatically generated by fitting the plane to the vanishing direction. A rendering of a three-dimensional model with the plane is displayed. Three-dimensional points corresponding to features common to the photos may be used to constrain the plane. The photos may be projected onto the model to provide visual feedback when editing the plane. Furthermore, the photos may be used to texture the model.

Claims

1. A computer implemented method of fitting a plane for a three-dimensional model, the model comprising vertices interconnected by line segments, the plane and model being represented by data in computer memory, the method comprising:

accessing;

a collection of digital photos and a cloud of points, where the photos are of a real world scene, where the photos are three-dimensionally arranged in a three-dimensional space containing the three-dimensional model, where the points are also arranged in the three-dimensional space, and where each point in the cloud corresponds to a feature of the scene that exists in at least two of the photos;

for each digital photo, for a given digital photo, computing and storing a vanishing direction specific to the given digital photo by analyzing the given digital photo to identify a corresponding vanishing point specific to the given digital photo, and using the vanishing point and the orientation of the given digital photo in the three-dimensional space to compute the vanishing direction specific to the given digital photo;

receiving user input and according to the user input selecting, from among the first and second vanishing directions, the first vanishing direction;

arranging the plane in the three-dimensional space by fitting the plane to at least the vanishing direction specific to the first digital photo and to at least one of the points in the plurality of points; and

displaying a rendering of the model according to the fitted plane.

2. A computer implemented method according to claim 1, further comprising:

displaying the first digital photo and superimposing onto the first digital photo points that correspond to points in the three-dimensional model, including the plurality of points; and

receiving the user input by the user selecting an area of the displayed first digital photo.

3. A computer implemented method according to claim 1, further comprising analyzing a second of the digital photos and using a line from the second digital photo to perform the fitting of the plane or to fit another plane.

4. A computer implemented method according to claim 1, wherein the vanishing point of the first digital photo is found by analyzing near-parallel lines in the first photo.

5. A computer implemented method according to claim 1, the process further comprising displaying one of the digital photos, receiving strokes directed to and displayed on the one of the digital photos, and using the strokes to identify parts of the one of the digital photos to use in a texture for the plane.

6. A computer implemented method according to claim 1, the process further comprising generating a texture image for the plane by using points in the cloud of points to automatically identify a portion of the first photo to include or omit in the texture image.

7. One or more volatile and/or non-volatile computer readable media storing information to enable a computer to perform a process, wherein the computer readable media is not a signal, the process comprising:

analyzing a first digital image of a first view of a real world scene to identify lines in the first view of the scene identifying thereamong a first set of lines that are determined to be parallel in the first view of the scene;

analyzing a second digital image of a second view of a real world scene to identify lines in the second view of the scene and identifying thereamong a second set of lines that are determined to be parallel in the second view of the scene;

determining a first three-dimensional vanishing direction in a three-dimensional space based on the first set of parallel lines and an orientation of the first digital image in the three-dimensional space, the determining using parallel lines only from the first digital image;

determining a second three-dimensional vanishing direction in the three-dimensional space based on the second set of parallel lines and an orientation of the second digital image in the three-dimensional space, the determining using parallel lines only from the second digital image;

automatically fitting a plane of a model in the three-dimensional space by determining which of the vanishing directions provides a more optimal fit of the plane to the three-dimensional model and then constraining the plane to both the determined one of vanishing directions and to a point in the three-dimensional space; and

displaying a rendering of a three-dimensional model in the three-dimensional space, where the model includes the fitted plane.

8. One or more computer readable media according to claim 7, the process further comprising:

accessing a plurality of points in the three-dimensional space, where each point comprises a respective feature in the scene that is common to a group of corresponding digital images in the three-dimensional space;

selecting the point from the plurality of points.

9. One or more computer readable media according to claim 8, further comprising displaying the first digital image, allowing a user to interactively define a region of interest in the first digital image, and the selecting the one of a plurality of points comprises selecting a point that maps to the region of interest defined by the user.

10. One or more computer readable media according to claim 9, wherein the region of interest comprises a polygon drawn by the user on the display of the first digital image, and where a side of the polygon is automatically aligned with the set of lines.

11. One or more computer readable media according to claim 7, wherein the first set of lines is computed by testing different pairs of lines in the first view, which groups lines into different sets of lines that are parallel in the real world scene.

12. One or more computer readable media according to claim 7, the process further comprising:

repeatedly re-projecting the first and second digital images of the real world scene onto the three-dimensional model while the orientation and/or position of the plane is repeatedly modified.

13. One or more computer readable media according to claim 7, the process further comprising translating the plane along the normal of the plane in accordance with user input.