Method and apparatus for automatically generating a site model

• US 7,509,241 B2
• Filed: 07/08/2002
• Issued: 03/24/2009
• Est. Priority Date: 07/06/2001
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Hsu, Stephen Charles, Kumar, Rakesh, Guo, Yanlin, Sawhney, Harpreet Singh
• Primary Examiner(s): Shah; Kamini S
• Assistant Examiner(s): Saxena; Akash
• Application Number:
  US10/191,397
• Publication Number:
  US 20030014224A1
• Time in Patent Office: 2,451 Days
• Field of Search: 345/418, 703/2, 703/7
• US Class Current: 703/2
• CPC Class Codes: G06F 30/13 Architectural design, e.g. computer-aided architectural design [CAAD] related to design of buildings, bridges, landscapes, production plants or roads
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris

Abstract

A method and apparatus for automatically combining aerial images and oblique images to form a three-dimensional (3D) site model. The apparatus or method is supplied with aerial and oblique imagery. The imagery is processed to identify building boundaries and outlines as well as to produce a depth map. The building boundaries and the depth map may be combined to form a 3D plan view model or used separately as a 2D plan view model. The imagery and plan view model is further processed to determine roof models for the buildings in the scene. The result is a 3D site model having buildings represented rectangular boxes with accurately defined roof shapes.

Claims

1. A method tot generating a three-dimensional site model of a site that contains objects comprising:

(a) providing at least one of aerial imagery and oblique imagery of the site;

(b) processing at least one of the aerial imagery and the oblique imagery to generate a plan view model comprising object outlines,

wherein said processing further comprises producing an elevation map using key video images;

analyzing the elevation map to identify surface patches;

computing local orientation of the surface patches within the elevation map, aggregating the surface patches for the key video images with respect to the local orientation of the surface patches;

aggregating elevation information amongst the key video images to produce a three-dimensional structure;

processing the three-dimensional structure into a plan view model; and

(c) producing from the plan view model a three-dimensional site model comprising blocks that represent objects within the site,

wherein the objects within the site include at least one building, the rectilinear outline of the at least one building being determined by;

computing an orientation histogram for the initial rectangle representing the building and selecting the angle of orientation with the strongest peak as the dominant orientation of the building;

computing a gradient field based on image intensity of the building;

integrating the gradient field along and perpendicular to the dominant orientation of the building;

fitting straight lines to local maximums of the gradient field along a each direction;

detecting knot points at the crossings of the straight lines;

dividing the initial rectangle into a plurality of smaller rectangles based on the knot points; and

traversing the smaller rectangles to find the rectilinear outline of the building.

2. The method of claim 1 further comprising:

(d) processing the oblique imagery to generate elevation information regarding the objects within the site.

3. The method of claim 1 wherein step (b) further comprises:

(b1) identifying object regions within the elevation map; and

(b2) processing the object regions by classifying color, features and texture within said object regions.

9. The method of claim 1 further comprising:

(d) deriving elevation information from the oblique imagery;

(e) sweeping a horizontal plane through a point cloud in the elevation information

(e1) identifying intersections of the point clouds with the horizontal plane to determine edges of the point density within the point cloud;

(f) processing a point density to identify object locations within the site.

10. The method of claim 9 wherein step (f) further comprises:

(f1) fitting rectangular edges of boxes to the point density, where the rectangular boxes represent objects within the site.

11. The method of claim 9 wherein step (f) further comprises:

(f1) analyzing the point density to identify object shapes and sizes.

12. The method of claim 11 wherein step (f1) further comprises:

(f11) sweeping at least one additional plane through the point cloud in the elevation information to produce point densities representing planes of objects within the site.

13. The method of claim 1 wherein the plan view model is produced using graphical analysis.

4. The method if claim wherein the objects in the site are buildings and the method further comprises:

(d) processing the oblique imagery to produce a roof model for each building in the site.

5. The method of claim 4 wherein step (d) further comprises:

(d1) selecting a building within the site;

(d2) detecting edges of the building within the oblique imagery;

(d3) identifying a set of roof model hypotheses that correspond to the edges; and

(d4) selecting and fitting each roof model hypothesis to the oblique imagery to identify a roof model hypothesis as the roof model.

6. The method of claim 5 wherein the step (d1) selects a building based upon visibility within the oblique imagery.

7. The method of claim 4 wherein step (d) further comprises:

(d1) selecting a building within the site;

(d2) extracting a range map from the oblique imagery;

(d3) mapping pixel locations within the range map to a common orthographic coordinate apparatus; and

(d4) extracting multiple planes from the range map using regression to fit the planes to form a roof model.

8. The method of clam 1 further comprising receiving elevation information from an elevation information source.

14. Apparatus for generating a three-dimensional site model of a site that contains objects comprising:

(a) means for providing at least one of aerial imagery and oblique imagery of the site;

(b) means for processing at least one of the aerial imagery or the oblique imagery to generate a plan view model comprising object outlines,

wherein processing fun her comprises means for producing an elevation map using key video images;

means for analyzing the elevation map to identify surface patches;

means for computing local orientation of the surface patches within the elevation map;

means for aggregating the surface patches for the key video images with respect to the local orientation of the surface patches;

means for aggregating elevation information amongst the key video images to produce a three-dimensional structure; and

means for processing the three-dimensional structure into a plan view model; and

(c) means for producing from the plan view a three-dimensional site model comprising blocks that represent objects within the site,

wherein the objects within the site include at least one building the rectilinear outline of the at least one building being determined by;

computing an orientation histogram for the initial rectangle representing the building and selecting the angle of orientation with die strongest peak as the dominant orientation of the building;

computing a gradient field based on image intensity of the building;

integrating the gradient field along and perpendicular to the dominant orientation of the building;

fitting straight lines to local maximums of the gradient field along each direction;

detecting knot points at the crossings of the straight lines;

dividing the initial rectangle into a plurality of smaller rectangles based on the knot points; and

traversing the smaller rectangles to find the rectilinear outline of the building.

15. The apparatus of claim 14 further comprising:

(d) means for processing the oblique imagery to generate depth information regarding the objects within the site.

16. The apparatus of claim 14 therein means (b) comprises:

(b1) means for identifying object regions within the elevation map; and

(b2) means for processing the object regions by classifying color, features and texture within said object regions.

17. The apparatus of claim 14 wherein the object in the site are buildings and the apparatus further comprises:

(d) means for processing the oblique imagery to produce a root model for each building in the site.

20. The apparatus of claim 17, wherein means (d) further comprises:

(d1) means for selecting a building within the site;

(d2) means for extracting a range map from the oblique imagery;

(d3) means for mapping pixel locations within the range map to a common orthographic coordinate apparatus;

(d4) means for extracting multiple planes from the range map using regression to fit the planes to form a roof model.

21. The apparatus of claim 14 further comprising an elevation information source.

22. The apparatus of claim 14 further comprising

(d) means for deriving range information from the oblique imagery;

(e) means for sweeping a horizontal plane through a point cloud in the range information;

(e1) means fry identifying intersections of the point clouds with the horizontal plane to determine edges of the point density within the point cloud;

(f) means for processing a point density to identify object locations within the site.

23. The apparatus of claim 22 wherein means (f) further comprises:

(f1) means for fitting rectangular boxes to edges of the point density where the rectangular boxes represent objects within the site.

24. The apparatus of claim 22 wherein means (f) further comprises:

(f1) means for analyzing the point density to identify object shapes and sizes.

25. The apparatus of claim 24 wherein means (f1) further comprises:

(f11) means for sweeping at least one additional plane through the point cloud in the range information to produce point densities representing planes of objects within the site.

26. the apparatus of claim 14 further comprising a means for producing a plan view model using graphical analysis.

18. The apparatus of clam 17 wherein means (d) further comprises:

(d1) means for selecting a building within the site;

(d2) means for detecting edges of the building within the oblique imagery;

(d3) means for identifying a set of roof model hypotheses that correspond to the edges;

(d4) means for selecting and fitting each roof model hypothesis to the oblique imagery to identify a roof model hypothesis as the roof model.

19. The apparatus of claim 18 wherein the means (d1) selects a building based upon visibility within the oblique imagery.