Registration of street-level imagery to 3D building models

• US 8,284,190 B2
• Filed: 06/25/2008
• Issued: 10/09/2012
• Est. Priority Date: 06/25/2008
• Status: Active Grant

• Current Assignee: Uber Technologies, Inc.
• Original Assignee: Microsoft Corporation
• Inventors: Muktinutalapati, Kartik Chandra, Tabb, Mark David, Nagy, Pete, Bi, Zhaoqiang, Kimchi, Gur
• Primary Examiner(s): HAJNIK, DANIEL F
• Application Number:
  US12/145,515
• Publication Number:
  US 20090322742A1
• Time in Patent Office: 1,567 Days
• Field of Search: None
• US Class Current: 345/419
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30184 Infrastructure
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/75 involving models

Abstract

Point of origin information for image data may be inaccurately registered against a geographic location absolute. A process for aligning image and highly accurate model data adjusts a point of origin of the image data by matching elements in the image with corresponding elements of the model. In a street-level image, building skylines can be extracted and corresponding skylines from the building model can be placed over the image-based skyline. By adjusting the point of origin of the image, the respective skylines can be aligned. Building edge and facade depth information can similarly be matched by adjusting the image point of origin of the image. The adjusted point of origin of the image can be used to then automatically place images on the models for a long run of images.

Claims

1. A method of correcting position data comprising acts of:

assigning a source location to an image;

determining a skyline detail of a building in the image;

extracting a building model corresponding to the skyline detail from a three dimensional scan;

projecting a model skyline detail from the building model onto the skyline detail from the image;

adjusting the source location to align the skyline detail and the model skyline detail in order to match elements in the image with corresponding elements of the model within a limit value, the adjusting the source location further comprises;

adjusting a start point location over a start point three-dimensional range defined by a three-dimensional cube; and

using a plurality of images to project a plurality of model details onto a corresponding plurality of building details from each image, where each respective projection is based on a given camera location in different cells of a matrix dividing the cube, the distance between the extracted skyline detail and projected model skyline is calculated as a sum of absolute distances in two dimensions in image coordinates of extracted skyline detail and projected model skyline, the adjusting performs the matching by using the distances as a measure;

recording an adjusted source location resulting from adjusting the source location, to determine a correction factor for the source location, for refining accuracy of geographic position and re-orienting an entire track of a run path comprising a plurality of image capture locations; and

utilizing a processor to execute instructions stored in memory to perform at least one of the acts of assigning, determining, extracting, projecting, adjusting, or recording.

2. The method of claim 1, further comprising:

using the adjusted source location when applying the image to the building model.

3. The method of claim 1, wherein determining the skyline detail in the image comprises:

measuring a distance to a facade of the building from the source location.

4. The method of claim 1, wherein determining the skyline detail in the image comprises:

evaluating edges and gradient magnitude;

evaluating pixels on a line associated with a vanishing point; and

estimating the skyline detail from the building model.

5. The method of claim 1, further comprising combining building depth data and the skyline detail when projecting the model detail.

6. The method of claim 1, wherein determining the skyline detail comprises determining a photographic building skyline.

7. The method of claim 1, wherein determining the skyline detail comprises determining a photographic building edge.

8. The method of claim 1, wherein determining the skyline detail comprises determining a LIDAR skyline.

9. The method of claim 1, wherein determining the skyline detail comprises determining a LIDAR building edge.

10. A method of aligning image data to a building model comprising acts of:

capturing street-level building image data information;

assigning a source location to the street-level building image data information;

determining a building skyline, building edge, and a building facade depth in the street-level building image data information;

extracting model information corresponding to a location associated with the street-level building image data information from aerial data;

generating a figure of merit function based on the distance between the building skyline, building edge, and building facade depth and corresponding elements of the model information;

applying the figure of merit function to the corresponding elements of the street-level building image data information and the aerial data for a plurality of locations along a street-scene and linearly combining the respective figures of merit for each of the plurality of locations;

calculating a displacement factor based on an analysis of an output of the figure of merit function;

modifying the source location by iteratively applying and recalculating the displacement factor to the source location;

aligning the street-level building image data information to the building model using the source location modified by the displacement factor; and

utilizing a processor to execute instructions stored in memory to perform at least one of the acts of capturing, assigning, determining, extracting, generating, applying, calculation, modifying or aligning.

11. The method of claim 10, wherein applying the figure of merit function comprises analyzing a street-level skyline and an aerial data skyline.

12. The method of claim 10, wherein applying the figure of merit function comprises analyzing a street-level facade and an aerial data facade.

13. The method of claim 10, wherein applying the figure of merit function comprises analyzing a street-level building edge and an aerial data building edge.

14. A system comprising a computer-readable memory having computer-executable instructions for executing a method comprising acts of:

capturing street-level building information including street-scene image data;

assigning a source location to the street-level building information;

determining a building skyline, building edge, and a building facade depth in the street-level building information;

extracting a building model from aerial data corresponding to a location associated with the street-level building information;

generating a figure of merit function based on a comparison of the building skyline, building edge, and building facade depth and corresponding elements of the building model;

applying the figure of merit function to the corresponding elements of the street-level building information and the building model based on the aerial data for a plurality of locations along a street-scene and linearly combining resulting respective figures of merit for each of the plurality of locations;

determining a displacement factor based on an analysis of an output of the figure of merit function;

modifying the source location by iteratively applying the displacement factor to the source location to refine accuracy of the source location;

aligning the street-scene image data to the building model using the source location modified by the displacement factor; and

a processor that executes the computer-executable instructions associated with at least one of the acts of capturing, assigning, determining, extracting, generating, applying, modifying, or aligning.

15. The computer-readable memory of claim 14, wherein applying the figure of merit function comprises comparing street-level building skyline, edge, and facade information with corresponding aerial data building skyline, edge, and facade information.

16. The computer-readable memory of claim 14, wherein capturing street-level building information comprises capturing street-level building information for both sides of a street.

17. The computer-readable memory of claim 14, wherein capturing street-level building information comprises capturing at least one of a two-dimensional photograph and three dimensional laser ranging (LIDAR) data.