REGISTRATION OF STREET-LEVEL IMAGERY TO 3D BUILDING MODELS

• US 20090322742A1
• Filed: 06/25/2008
• Published: 12/31/2009
• Est. Priority Date: 06/25/2008
• Status: Active Grant

• Current Assignee: Uber Technologies, Inc.
• Original Assignee: Microsoft Corporation
• Inventors: Muktinutalapati, Kartik Chandra, Tabb, Mark David, Nagy, Pete, Bi, Zhaoqiang, Kimchi, Gur
• Granted Patent: US 8,284,190 B2
• Time in Patent Office: Days
• US Class Current: 345/420
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30184 Infrastructure
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/75 involving models

Abstract

Point of origin information for image data may be inaccurately registered against a geographic location absolute. A process for aligning image and highly accurate model data adjusts a point of origin of the image data by matching elements in the image with corresponding elements of the model. In a street-level image, building skylines can be extracted and corresponding skylines from the building model can be placed over the image-based skyline. By adjusting the point of origin of the image, the respective skylines can be aligned. Building edge and facade depth information can similarly be matched by adjusting the image point of origin of the image. The adjusted point of origin of the image can be used to then automatically place images on the models for a long run of images.

Claims

1. A method of correcting position data comprising:

assigning a source location to an image;

determining a building detail of a building in the image;

extracting a building model corresponding to the building detail from a three dimensional scan;

projecting a model detail from the building model onto the building detail from the image;

adjusting the source location until the building detail and the model detail match within a limit value;

recording an adjusted source location resulting from adjusting the source location until the building detail and the model detail match within the limit value.

2. The method of claim 1, further comprising:

using the adjusted source location when applying the image to the building model.

3. The method of claim 1, wherein determining the building detail in the image comprises:

measuring a distance to a facade of the building from the source location.

4. The method of claim 1, wherein determining the building detail in the image comprises:

evaluating edges and gradient magnitude;

evaluating pixels on a line associated with a vanishing point; and

estimating the building detail from the building model.

5. The method of claim 1, further comprising selecting a plurality of images corresponding to buildings from a run segment and from opposing sides of the run segment.

6. The method of claim 5, further comprising combining building depth data and building skyline data when projection the model detail.

7. The method of claim 1, wherein determining the building detail comprises determining a photographic building skyline.

8. The method of claim 1, wherein determining the building detail comprises determining a photographic building edge.

9. The method of claim 1, wherein determining the building detail comprises determining a LIDAR skyline.

10. The method of claim 1, wherein determining the building detail comprises determining a LIDAR building edge.

11. The method of claim 1, wherein adjusting the source location comprises:

adjusting a start point source location over a start point three dimensional range using a plurality of images to project a plurality of model details onto a corresponding plurality of building details from each image.

12. A method of aligning image data to a building model comprising:

capturing street-level building information;

assigning a source location to the street-level building information;

determining a building skyline, building edge, and a building facade depth in the street-level building information;

extracting model information corresponding to a location associated with the street-level building information from aerial data;

applying a figure of merit function to corresponding elements of the street-level building information and the aerial data;

calculating a displacement factor based on an analysis of an output of the figure of merit function;

modifying the source location by applying the displacement factor to the source location; and

aligning the image data to the building model using the source location modified by the displacement factor.

13. The method of claim 12, wherein applying the figure of merit function comprises analyzing a street-level skyline and an aerial data skyline.

14. The method of claim 12, wherein applying the figure of merit function comprises analyzing a street-level facade and an aerial data facade.

15. The method of claim 12, wherein applying the figure of merit function comprises analyzing a street-level building edge and an aerial data building edge.

16. A computer-readable storage medium having computer-executable instructions for executing a method comprising:

capturing street-level building information including street-scene image data;

assigning a source location to the street-level building information;

determining a building skyline, building edge, and a building facade depth in the street-level building information;

extracting a building model from aerial data corresponding to a location associated with the street-level building information;

applying a figure of merit function to corresponding elements of the street-level building information and the building model based on the aerial data;

determining a displacement factor based on an analysis of an output of the figure of merit function;

modifying the source location by applying the displacement factor to the source location; and

aligning the street-scene image data to the building model using the source location modified by the displacement factor.

17. The method of claim 16, wherein applying the figure of merit function comprises comparing street-level building skyline, edge, and facade information with corresponding aerial data building skyline, edge, and facade information.

18. The method of claim 16, wherein capturing street-level building information comprises capturing street-level building information for both sides of a street.

19. The method of claim 16, wherein capturing street-level building information comprises capturing at least one of a two-dimensional photograph and three dimensional laser ranging (LIDAR) data.

20. The method of claim 16, wherein applying a figure of merit function to corresponding elements of the street-level building information and the building model based on the aerial data comprises applying a figure of merit function to corresponding elements of the street-level building information and the building model based on the aerial data for a plurality of locations along a street-scene and linearly combining the respective figures of merit for each of the plurality of locations.