Map-like summary visualization of street-level distance data and panorama data

• US 9,984,494 B2
• Filed: 01/26/2015
• Issued: 05/29/2018
• Est. Priority Date: 01/26/2015
• Status: Active Grant

• Current Assignee: Uber Technologies, Inc.
• Original Assignee: Uber Technologies, Inc.
• Inventors: Pylvaenaeinen, Timo, Korah, Thommen, Berclaz, Jerome, Nam, Myra
• Primary Examiner(s): NGUYEN, ANH TUAN V
• Application Number:
  US14/605,912
• Publication Number:
  US 20160217611A1
• Time in Patent Office: 1,219 Days
• Field of Search: None
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 15/405 using Z-buffer
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 3/04 Context-preserving transformations, e.g. by using an importance map panospheric to cylindrical image transformations G06T3/12
• CPC Class Codes: G06T 3/053 Detail-in-context presentations fisheye or wide-angle transformations G06T3/047
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Architecture that summarizes a large amount (e.g., thousands of miles) of street-level image/video data of different perspectives and types (e.g., continuous scan-type data and panorama-type data) into a single view that resembles aerial imagery. Polygons surfaces are generated from the scan patterns and the image data is projected onto the surfaces, and then rendered into the desired orthographic projection. The street-level data is processed using a distributed computing approach across cluster nodes. The collection is processed into image tiles on the separate cluster nodes representing an orthographic map projection that can be viewed at various levels of detail. Map features such as lower-level roads, that are at lower elevations than higher-level roads, and are hidden by higher-level overpassing roads, can be navigated in the map. With the summarized data, the maps can be navigated and zoomed efficiently.

Claims

1. A system to generate a three-dimensional oblique aerial map view, comprising:

one or more processors;

a non-transitory computer readable storage medium comprising instructions, that when executed by the one or more processors, cause the one or more processors to;

receive distance data to points in a street-level scene and panorama images that include the street-level scene, the distance data sensed using a LIDAR scanning system and the panorama images captured using an image-capture system;

responsive to a determination that a first scan data point in the distance data has a missing pixel value in the panorama images;

interpolate a pixel value for the missing pixel value based at least on pixel data of the panorama images that correspond to scan data points adjacent in location to the first scan data point;

substitute the interpolated pixel value for the missing pixel value;

distributively process the distance data and the panorama images using distributed computing nodes, each computing node configured to process an arbitrary set of the panorama images into image tiles and the distance data into associated depth maps of specified orthographic projections;

merge two or more partial renderings of an image tile of a same area into a single rendering of the image tile using z-buffer values of associated depth maps by retaining pixel values in the two or more partial renderings that have a highest z-buffer value of the associated depth maps, each of the two or more partial renderings of the image tiles rendered independently from different panorama images; and

generate a three-dimensional oblique aerial viewpoint based on the image tiles.

2. The system of claim 1, the non-transitory computer readable storage medium comprising further instructions, that when executed by the processor, cause the processor to:

generate polygons of a specific geometry based on a scan pattern of the scanning system.

3. The system of claim 2, the non-transitory computer readable storage medium comprising further instructions, that when executed by the processor, cause the processor to:

project the panorama images onto the polygons to create projected images as the specified orthographic projections.

4. The system of claim 2, the non-transitory computer readable storage medium comprising further instructions, that when executed by the processor, cause the processor to:

generate depth maps of the polygons onto a z-buffer as the depth maps of the specified orthographic projections.

5. The system of claim 1, wherein the distributed computing nodes each processes a specific arbitrary set of the distance data, and one or more of the panorama images associated only with the specific arbitrary set.

6. A method, comprising:

receiving distance data to points of a street-level scene and panorama data of the street-level scene, the distance data sensed using a LIDAR scanning system and the panorama data captured using an image-capture system;

partitioning the distance data into distance datasets that correspond to specific portions of the street-level scene;

accessing panorama data related to a selected distance dataset;

responsive to determining that a first scan data point in the selected distance dataset has a missing pixel value in the panorama data;

interpolating a pixel value for the missing pixel value based at least on pixel data of the panorama data that correspond to scan data points adjacent in location to the first scan data point; and

substituting the interpolated pixel value for the missing pixel value;

distributively processing the selected distance dataset and panorama data on corresponding distributed processing nodes to create interim tiles, each computing node configured to process an arbitrary set of the panorama data into interim tiles and the distance dataset into associated depth maps;

merging two or more partial renderings of an interim tile of a same area into a single rendering of the interim tile using z-buffer values of associated depth maps by retaining pixel values in the two or more partial renderings that have a highest z-buffer value of the associated depth maps, each of the two or more partial renderings of the interim tiles rendered independently from different panorama data;

merging the interim tiles from each of the distributed processing nodes into a final tile; and

projecting the final tile to generate an aerial viewpoint.

7. The method of claim 6, further comprising adjusting the aerial viewpoint using various levels of view adjustment.

8. The method of claim 6, further comprising receiving the distance data from a multi-head distance acquisition scanning system.

9. The method of claim 8, further comprising generating polygons based on a scan pattern of the scanning system, the polygons having a specific geometry, and projecting imagery onto the geometry.

10. The method of claim 8, further comprising choosing a closer point that is closer to the scanning system when compared to a prior farther point that is farther from the scanning system.

11. The method of claim 6, further comprising providing visualization of multiple elevation layers of road maps to enable navigation of roads visually hidden by overpassing roads.

12. A method, comprising acts of:

receiving distance data to physical points of a street-level scene, the distance data sensed using a LIDAR scanning system;

partitioning the distance data into distance datasets;

selecting panorama data for each of the distance datasets, the panorama data captured using an image-capture system;

responsive to determining that a first scan data point in one of the distance dataset has a missing pixel value in the corresponding panorama data;

interpolating a pixel value for the missing pixel value based at least on pixel data in the panorama data that correspond to scan data points adjacent in location to the first scan data point; and

substituting the interpolated pixel value for the missing pixel value;

distributively processing each distance dataset and corresponding panorama data independently in accordance with corresponding distributed processing nodes, wherein two or more renderings of each portion of panorama data of a same area are merged using the z-buffer values of corresponding distance dataset by retaining pixel values in the two or more renderings that have the highest z-buffer value of the corresponding distance dataset, each of the two or more renderings rendered independently from different panorama data;

receiving node results from each of the distributed processing nodes;

merging the node results into a final result; and

projecting the final result into a tile that provides an aerial viewpoint.

13. The method of claim 12, further comprising removing visual clutter from a route using road surface estimation.

14. The method of claim 12, further comprising generating polygons based on a LID AR scan pattern.

15. The method of claim 12, further comprising visualizing multiple elevations of layers of map features.

16. The method of claim 12, further comprising representing tiles as an orthographic map projection that enables aerial viewing at various levels of detail and from multiple points of view and elevations.