Spatially registering user photographs

• US 8,611,643 B2
• Filed: 09/14/2012
• Issued: 12/17/2013
• Est. Priority Date: 05/20/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ofek, Eyal, Hou, Dan, Aguera y Arcas, Blaise, Fynn, Scott Vernon, Molinari, Roberto R., Kroepfl, Michael, Ernst, Tony
• Primary Examiner(s): Shah, Uptal
• Application Number:
  US13/618,351
• Publication Number:
  US 20130009954A1
• Time in Patent Office: 459 Days
• Field of Search: 382/154, 382/294, 382296-298
• US Class Current: 382/154
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling

Abstract

Photographs of an object may be oriented with respect to both the geographic location and orientation of the object by registering a 3D model derived from a plurality of photographs of the objects with a 2D image of the object having a known location and orientation. For example, a 3D point cloud of an object created from photographs of the object using a Photosynth™ tool may be aligned with a satellite photograph of the object, where the satellite photograph has location and orientation information. A tool providing scaling and rotation of the 3D model with respect to the 2D image may be used or an automatic alignment may be performed using a function based on object edges filtered at particular angles. Once aligned, data may be recorded that registers camera locations for the plurality of photographs with geographic coordinates of the object, either absolute latitude/longitude or relative to the object.

Claims

1. A computer-implemented method for registering a three-dimensional (3D) representation of an object, the method comprising:

displaying the 3D representation and a generally overhead view of the object on a computer;

adjusting the displayed 3D representation to at least approximately match the generally overhead view of the object in scale, orientation, and two-dimensional (2D) translation, wherein adjusting the 3D representation includes orienting the 3D representation to have an approximately equal elevation angle as the generally overhead view of the object; and

associating a geographic coordinate of the generally overhead view of the object with the adjusted 3D representation, the geographic coordinate including at least one of a location relative to the object, an earth coordinate location of the object, an earth coordinate orientation of the object, or a magnetic coordinate orientation of the object.

2. The method of claim 1 wherein adjusting the displayed 3D representation includes performing at least one of:

scaling the 3D representation to approximately match a size of the object in the generally overhead view of the object;

or

rotating the 3D representation to approximately match an orientation of the object in the generally overhead view of the object.

3. The method of claim 1 wherein the 3D representation of the object is generated from a plurality of photographs of the object, the photographs being captured from individual camera locations relative to the object, and wherein the method further includes determining the individual camera locations based on the geographic coordinate of the generally overhead view of the object.

4. The method of claim 1 wherein the generally overhead view of the object is generally perpendicular to ground, and wherein displaying the 3D representation includes orienting the 3D representation to present a top view of the 3D representation of the object.

5. The method of claim 1 wherein the generally overhead view of the object is generally perpendicular to ground, and wherein:

displaying the 3D representation includes orienting the 3D representation to present a top view of the 3D representation of the object; and

adjusting the 3D representation includes performing at least one of

scaling the 3D representation to approximately match a size of the object in the generally overhead view of the object, or

rotating the 3D representation to approximately match an orientation of the object in the generally overhead view of the object.

6. The method of claim 1 wherein the generally overhead view includes a perspective view of the object taken with a camera at an altitude, and wherein adjusting the 3D representation includes translating a vertical projection of the 3D representation to be at an elevation approximately as the perspective view of the object taken from the altitude.

7. The method of claim 1 wherein the generally overhead view of the object has a geographic location and orientation, and wherein associating the geographic coordinate includes associating the geographic location and orientation of the generally overhead view of the object with the 3D representation of the object.

8. The method of claim 1 wherein the generally overhead view of the object has a latitude, a longitude, and a compass direction, and wherein associating the geographic coordinate includes associating the latitude, longitude, and compass direction of the generally overhead view of the object to the 3D representation of the object.

9. The method of claim 1 wherein the generally overhead view includes a perspective view of the object taken with a camera at an altitude, and wherein:

adjusting the 3D representation includes;

translating a vertical projection of the 3D representation to be at an elevation approximately as the perspective view of the object taken from the altitude;

performing at least one of scaling the 3D representation to approximately match a size of the object in the perspective view, or rotating the 3D representation to approximately match an orientation of the object in the perspective view; and

the method further includes calculating a ground height of the 3D representation based on the altitude of the camera and the perspective view of the object.

10. A computer-implemented method for spatially registering a three-dimensional (3D) representation of an object, the method comprising:

displaying the 3D representation and a generally overhead view of the object, the generally overhead view being at a perspective angle with ground;

performing at least one of rotating, scaling, or aligning the displayed 3D representation to

at least approximately match the displayed generally overhead view such that a vertical projection of the 3D representation is at an elevation having a perspective angle with ground that is approximately equal to the perspective angle of the generally overhead view; and

associating a geographic coordinate of the generally overhead view of the object to the rotated, scaled, or aligned 3D representation.

11. The method of claim 10 wherein displaying the 3D representation and the generally overhead view of the object includes displaying at least one of a map, an aerial photo, or a satellite photo of the object.

12. The method of claim 10 wherein displaying the 3D representation and the generally overhead view of the object includes displaying a two-dimensional (2D) photograph of the object taken with a camera having a frusta and a camera location relative to the object.

13. The method of claim 10 wherein displaying the 3D representation and the generally overhead view of the object includes displaying a top view of the 3D representation and a two-dimensional (2D) photograph of the object taken with a camera having a frusta and a camera location relative to the object.

14. The method of claim 10 wherein the method further includes calculating a ground height of the 3D representation based on an altitude of a camera used to take the generally overhead view of the object.

15. The method of claim 10 further comprising generating contrast regions for the 3D representation, the contrast regions individually having a lower opacity at an edge than at a center of the contrast regions.

16. The method of claim 10 further comprising decreasing visibility of the generally overhead view when performing at least one of rotating, scaling, or aligning the displayed 3D representation.

18. The computer-storage device of claim 16 wherein the 3D representation includes a plurality of feature points, and wherein determining the best fit includes generating a weighted score for the individual feature points at a plurality of scales, rotations, and 2D translations, and determining a best fit of the 3D representation at a particular one of the scales, rotations, and 2D translations based on the weighted scores.

19. The computer-storage device of claim 16 wherein filtering the generated edge map includes:

defining edge pixels of objects in the generated edge map; and

removing edge pixels of objects not lying along a particular direction.

20. The computer-storage device of claim 16 wherein the 3D representation includes a plurality of feature points, and wherein:

filtering the generated edge map includes;

defining edge pixels of objects in the generated edge map; and

removing edge pixels of objects not lying along a particular direction determining the best fit includes;

creating a distance map for the filtered edge map by computing for each pixel a distance to closest edge pixels in the filtered edge map;

computing a convolution of the distance map for edge pixels from the 2D image with the corresponding feature points of the 3D representation to generate a weighted score; and

iterating the convolution at a plurality of scales, rotations, and 2D translations to determine the best fit of the 3D representation with the 2D image.

17. A computer-storage device containing instructions that cause a processor to perform a method for registering a three-dimensional (3D) representation of an object, the method comprising:

generating an edge map showing edges of objects based on a two-dimensional (2D) image of the objects, the image having an associated geographic coordinate;

filtering the generated edge map to generate a filtered edge map having edges extending along generally the same direction;

determining a best fit of the 3D representation at a particular one of the scales, rotations, and 2D translations based on the filtered edge map of the 2D image; and

associating the geographic coordinate of the 2D image to the determined best fit of the 3D representation.

21. A computer-implemented method for registering a three-dimensional (3D) representation of an object, the method comprising:

displaying the 3D representation and a generally overhead view of the object on a computer;

generating contrast regions for the 3D representation displayed on the computer, the contrast regions individually having a lower opacity at an edge than at a center of the contrast regions;

performing at least one of rotating, scaling, or aligning the displayed 3D representation to at least approximately match the displayed generally overhead view of the object; and

associating a geographic coordinate of the generally overhead view of the object to the rotated, scaled, or aligned 3D representation.

22. The method of claim 21 wherein displaying the 3D representation and the generally overhead view of the object includes displaying at least one of a map, an aerial photo, or a satellite photo of the object.

23. The method of claim 21 wherein displaying the 3D representation and the generally overhead view of the object includes displaying a two-dimensional (2D) photograph of the object taken with a camera having a frusta and a camera location relative to the object.

24. The method of claim 21 wherein displaying the 3D representation and the generally overhead view of the object includes displaying a top view of the 3D representation and a two-dimensional (2D) photograph of the object taken with a camera having a frusta and a camera location relative to the object.

25. The method of claim 21 wherein a vertical projection of the 3D representation is at an elevation approximately as a perspective view of the object, and wherein the method further includes calculating a ground height of the 3D representation based on an altitude of a camera used to take the generally overhead view of the object.

26. The method of claim 21 further comprising decreasing visibility of the generally overhead view when performing at least one of rotating, scaling, or aligning the displayed 3D representation.