Spatially Registering User Photographs

• US 20110286660A1
• Filed: 05/20/2010
• Published: 11/24/2011
• Est. Priority Date: 05/20/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ernst, Tony, Hou, Dan, Molinari, Roberto R., Fynn, Scott Vernon, Arcas, Blaise Aguera y, Ofek, Eyal, Kroepfl, Michael
• Granted Patent: US 8,295,589 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling

Abstract

Photographs of an object may be oriented with respect to both the geographic location and orientation of the object by registering a 3D model derived from a plurality of photographs of the objects with a 2D image of the object having a known location and orientation. For example, a 3D point cloud of an object created from photographs of the object using a Photosynth™ tool may be aligned with a satellite photograph of the object, where the satellite photograph has location and orientation information. A tool providing scaling and rotation of the 3D model with respect to the 2D image may be used or an automatic alignment may be performed using a function based on object edges filtered at particular angles. Once aligned, data may be recorded that registers camera locations for the plurality of photographs with geographic coordinates of the object, either absolute latitude/longitude or relative to the object.

Claims

1. A method of registering a three dimensional (3D) representation of an object with an oriented view of the object on a computer comprising:

providing the 3D representation of the object generated at least in part from a plurality of photographs of the object, each of the plurality of photographs having a respective camera location calculated relative to the 3D representation of the object;

providing a generally overhead view of the object;

orienting the 3D representation to have an approximately equal elevation angle as the generally overhead view;

scaling the 3D representation to approximately match a size of the object in the generally overhead view of the object;

rotating the 3D representation to approximately align with an orientation of the object in the generally overhead view of the object;

aligning the 3D representation with the object in the generally overhead view of the object;

registering each of the respective camera locations with the generally overhead view of the object; and

storing registration data for the 3D representation of the object corresponding to at least one of a location relative to the object, an earth coordinate location, an earth coordinate orientation, and a magnetic coordinate orientation of the 3D representation, wherein the 3D representation of the object is a point cloud being a set of points in 3D space that collectively make up the 3D representation, the method further comprising providing a tool for performing the scaling and rotating of the 3D representation, the tool having at least one of a first control for scaling the 3D representation, a second control for rotating the 3D representation, and a third control for resizing the tool, wherein providing the 3D representation of the object comprises creating a flattened view of the point cloud perpendicularly to the approximately equal elevation angle.

2. The method of claim 1, further comprising generating a contrast region around each point in the flattened view.

3. The method of claim 2, wherein generating the contrast region comprises generating the contrast region with a decreased opacity at an edge of the contrast region compared to an opacity at a center of the contrast region.

4. The method of claim 1, wherein aligning the 3D representation with the generally overhead view of the object comprises moving the generally overhead view of the object while the tool remains centered in a viewing area.

5. The method of claim 1, wherein providing the tool comprises providing the tool to decrease visibility of the generally overhead view when the tool is active for aligning, scaling, or rotating to provide increased visibility of the 3D representation when being operated on.

6. The method of claim 1, further comprising presenting at least one of the plurality of photographs and its associated camera frusta showing the at least one of the photograph'"'"'s respective calculated camera location oriented with the 3D representation.

7. The method of claim 1, wherein the scaling, rotating, and aligning steps are first performed manually using the tool and repeated using an automated process.

8. The method of claim 7, further comprising:

extracting from the plurality of photographs a color for areas visible from a top view of the object, wherein aligning the 3D representation comprises color matching pixels from the generally overhead view to the extracted color for areas visible from the top view.

9. The method of claim 1, further comprising:

providing a first perspective view of the object, the first perspective view captured from a known location;

generating a second perspective view of the 3D representation at a similar elevation of first perspective view;

registering the first and second perspective views; and

determining an absolute elevation of the 3D representation based on the known location.

10. A method of providing a tool for manually aligning a three dimensional (3D) point cloud representing a physical object with a view of the physical object implemented by computer-executable instructions stored on a computer-readable storage medium executed on a processor of a computer, the method comprising:

providing the view of the physical object, the physical object having a known location and the view of the physical object captured from a known perspective;

creating an initial view of the 3D point cloud from approximately the same perspective as the view of the physical object;

displaying the initial view of the 3D point cloud and the view of the physical object in a common window;

displaying the tool having controls affecting a scale and a rotation of the 3D point cloud;

providing a contrast enhancement of the 3D point cloud;

providing for a translation of the physical object with respect to the view of the 3D point cloud; and

storing the known location and an absolute orientation of the 3D representation with respect to the view of the physical object after matching the scale, the rotation, and the translation of the 3D representation with the view of the physical object.

11. The method of claim 10, further comprising displaying a two-dimensional (2D) photograph of the physical object including a camera frusta indicating a camera location associated with the 2D photograph, the 2D photograph oriented to the 3D point cloud in line with the camera frusta.

12. The method of claim 10, wherein providing the contrast enhancement of the 3D point cloud comprises increasing the contrast enhancement while the physical object is being translated with respect to the view of the 3D point cloud.

13. A method of automatically registering a 3D representation with a 2D image, each having common subject matter, comprising:

defining edge pixels (edgels) of objects in the 2D image;

removing edgels of objects not lying on predetermined angles;

creating a distance image for each edge image, by computing for each pixel a distance to its closest edgel in the edge image;

convolving, using a Fast Fourier Transform, the distance image for corresponding edgels from the 2D image with points from the 3D model, respectively to create a fit score;

iterating the convolving for different 3D model configurations to determine a best fit of the 3D model with the 2D image;

associating location and orientation information from the 2D image with the 3D model after reaching the best fit of 3D model with the 2D image.

14. The method of claim 13, wherein the removing edgels of objects not lying on predetermined angles comprises:

creating sets of edgels lying within about 10 degrees of lines lying at 45 degree intervals from an orientation axis.