Alignment of data captured by autonomous vehicles to generate high definition maps

• US 10,222,211 B2
• Filed: 12/28/2017
• Issued: 03/05/2019
• Est. Priority Date: 12/30/2016
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: DeepMap, Inc. (NVIDIA Corporation)
• Inventors: Chen, Chen, Adachi, Jeffrey Minoru
• Primary Examiner(s): Antonucci, Anne M
• Assistant Examiner(s): Stroud, James E
• Application Number:
  US15/857,602
• Publication Number:
  US 20180188039A1
• Time in Patent Office: 432 Days
• Field of Search: 701436
• CPC Class Codes: B60W 2552/00 Input parameters relating to infrastructure
• CPC Class Codes: B60W 2552/53 Road markings, e.g. lane marker or crosswalk
• CPC Class Codes: B60W 40/06 Road conditions
• CPC Class Codes: G01C 11/06 by comparison of two or more pictures of the same area
• CPC Class Codes: G01C 11/12 the pictures being supported in the same relative position as when they were taken
• CPC Class Codes: G01C 11/30 by triangulation
• CPC Class Codes: G01C 21/005 with correlation of navigation data from several sources, e.g. map or contour matching G01C21/30 takes precedence
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 21/3635 Guidance using 3D or perspective road maps
• CPC Class Codes: G01C 21/3694 Output thereof on a road map
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G01C 21/3867 Geometry of map features, e.g. shape points, polygons or for simplified maps
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 19/42 Determining position
• CPC Class Codes: G01S 19/46 the supplementary measurement being of a radio-wave signal type
• CPC Class Codes: G01S 19/47 the supplementary measurement being an inertial measurement, e.g. tightly coupled inertial
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10021 : Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 : Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20048 : Transform domain processing
• CPC Class Codes: G06T 2207/30252 : Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2207/30256 : Lane; Road marking
• CPC Class Codes: G06T 2210/56 : Particle system, point based geometry or rendering
• CPC Class Codes: G06T 2215/12 : Shadow map, environment map
• CPC Class Codes: G06T 7/11 : Region-based segmentation
• CPC Class Codes: G06T 7/246 : using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 : involving reference images or patches
• CPC Class Codes: G06T 7/55 : from multiple images
• CPC Class Codes: G06T 7/593 : from stereo images
• CPC Class Codes: G06T 7/68 : of symmetry
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06T 7/74 : involving reference images or patches
• CPC Class Codes: G06V 10/462 : Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/758 : Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 : Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G08G 1/20 : Monitoring the location of vehicles belonging to a group, e.g. fleet of vehicles, countable or determined number of vehicles

Abstract

A high-definition map system receives sensor data from vehicles travelling along routes and combines the data to generate a high definition map for use in driving vehicles, for example, for guiding autonomous vehicles. A pose graph is built from the collected data, each pose representing location and orientation of a vehicle. The pose graph is optimized to minimize constraints between poses. Points associated with surface are assigned a confidence measure determined using a measure of hardness/softness of the surface. A machine-learning-based result filter detects bad alignment results and prevents them from being entered in the subsequent global pose optimization. The alignment framework is parallelizable for execution using a parallel/distributed architecture. Alignment hot spots are detected for further verification and improvement. The system supports incremental updates, thereby allowing refinements of sub-graphs for incrementally improving the high-definition map for keeping it up to date.

Claims

1. A method for generating high definition maps for use in the driving of autonomous vehicles, the method comprising:

receiving sensor data captured by a plurality of vehicles driving through a path in a geographical region;

generating a pose graph, wherein each node of the pose graph represents a pose of a vehicle, the pose comprising a location and orientation of the vehicle, and wherein each edge between a pair of nodes represents a transformation between nodes of the pair of nodes;

selecting a subset of nodes from the pose graph, wherein selecting the subset of nodes from the pose graph comprises;

identifying nodes having high quality global navigation satellite system (GNSS) poses; and

increasing the likelihood of selecting the identified nodes compared to nodes with lower quality GNSS poses;

for each node of the subset of nodes, identifying a GNSS pose corresponding to the node;

performing optimization of the pose graph based on constraints that minimize the pose difference between each of the nodes of the subset and the corresponding GNSS pose;

merging sensor data captured by plurality of autonomous vehicles to generate a point cloud representation of a geographical region;

generating a high-definition map based on the point cloud representation; and

sending the high-definition map to one or more autonomous vehicles for navigating in the geographical region.

2. The method of claim 1, wherein the sensor data comprises LIDAR range images.

3. The method of claim 1, further comprising:

performing pairwise alignment between pairs of nodes associated with a single track, wherein a track represents a collection of data corresponding to a single drive of a vehicle through a route, the data comprising a sequence of poses with associated sensor data.

4. The method of claim 3, further comprising:

performing pairwise alignment between pairs of nodes, wherein a first node of the pair is selected from a first track and a second node of the pair is selected from a second track.

5. The method of claim 4, further comprising:

performing filtering of results of pairwise alignments to determine whether a result needs further review.

6. The method of claim 5, wherein the filtering of the results of pairwise alignments is performed using a machine learning based model configured to predict a score indicating whether a pairwise alignment result needs further review.

7. The method of claim 1, further comprising:

removing one or more samples of sensor data responsive to determining that the samples were captured by a stationary vehicle.

8. The method of claim 1, wherein selecting the subset of nodes from the pose graph comprises selecting nodes at fixed distance intervals along a path traveled by an autonomous vehicle capturing the sensor data corresponding to the node.

9. The method of claim 1, wherein selecting the subset of nodes from the pose graph comprises a random selection of nodes from the pose graph.

10. The method of claim 1, wherein selecting the subset of nodes from the pose graph comprises:

determining bounding boxes based on latitude and longitude values; and

selecting a node from each bounding box.

11. The method of claim 1, wherein selecting the subset of nodes from the pose graph comprises:

identifying a region having low quality of pairwise alignment; and

increasing the sampling frequency in the identified region.

12. A non-transitory computer readable storage medium storing instructions for:

receiving sensor data captured by a plurality of vehicles driving through a path in a geographical region;

generating a pose graph, wherein each node of the pose graph represents a pose of a vehicle, the pose comprising a location and orientation of the vehicle, and wherein each edge between a pair of nodes represents a transformation between nodes of the pair of nodes;

selecting a subset of nodes from the pose graph, wherein selecting the subset of nodes from the pose graph comprises;

identifying nodes having high quality global navigation satellite system (GNSS) poses; and

increasing the likelihood of selecting the identified nodes compared to nodes with lower quality GNSS poses;

for each node of the subset of nodes, identifying a GNSS pose corresponding to the node;

performing optimization of the pose graph based on constraints that minimize the pose difference between each of the nodes of the subset and the corresponding GNSS pose;

merging sensor data captured by plurality of autonomous vehicles to generate a point cloud representation of a geographical region;

generating a high-definition map based on the point cloud representation; and

sending the high-definition map to one or more autonomous vehicles for navigating in the geographical region.

13. The non-transitory computer readable storage medium of claim 12, wherein the stored instructions are for further:

performing pairwise alignment between pairs of nodes associated with a single track, wherein a track represents a collection of data corresponding to a single drive of a vehicle through a route, the data comprising a sequence of poses with associated sensor data.

14. The non-transitory computer readable storage medium of claim 13, wherein the stored instructions are for further:

performing pairwise alignment between pairs of nodes, wherein a first node of the pair is selected from a first track and a second node of the pair is selected from a second track.

15. The non-transitory computer readable storage medium of claim 14, wherein the stored instructions are for further:

performing filtering of results of pairwise alignments to determine whether a result needs further review, wherein the filtering of the results of pairwise alignments is performed using machine learning based model configured to predict a score indicating whether a pairwise alignment result needs further review.

16. The non-transitory computer readable storage medium of claim 12, wherein the stored instructions are for further:

removing one or more samples of sensor data responsive to determining that the samples were captured by a stationary vehicle.

17. A computer system comprising:

an electronic processor; and

a non-transitory computer readable storage medium storing instructions executable by the electronic processor, the instructions for;

receiving sensor data captured by a plurality of vehicles driving through a path in a geographical region;

generating a pose graph, wherein each node of the pose graph represents a pose of a vehicle, the pose comprising a location and orientation of the vehicle, and wherein each edge between a pair of nodes represents a transformation between nodes of the pair of nodes;

selecting a subset of nodes from the pose graph, wherein selecting the subset of nodes from the pose graph comprises;

identifying nodes having high quality global navigation satellite system (GNSS) poses; and

increasing the likelihood of selecting the identified nodes compared to nodes with lower quality GNSS poses;

for each node of the subset of nodes, identifying a GNSS pose corresponding to the node;

performing optimization of the pose graph based on constraints that minimize the pose difference between each of the nodes of the subset and the corresponding GNSS pose;

merging sensor data captured by plurality of autonomous vehicles to generate a point cloud representation of a geographical region;

generating a high-definition map based on the point cloud representation; and

sending the high-definition map to one or more autonomous vehicles for navigating in the geographical region.

18. The computer system of claim 17, wherein the stored instructions are for further:

performing pairwise alignment between pairs of nodes associated with a single track, wherein a track represents a collection of data corresponding to a single drive of a vehicle through a route, the data comprising a sequence of poses with associated sensor data.

19. The computer system of claim 18, wherein the stored instructions are for further:

performing pairwise alignment between pairs of nodes, wherein a first node of the pair is selected from a first track and a second node of the pair is selected from a second track.