Encoding LiDAR scanned data for generating high definition maps for autonomous vehicles

• US 10,401,500 B2
• Filed: 12/28/2017
• Issued: 09/03/2019
• Est. Priority Date: 12/30/2016
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: DeepMap, Inc. (NVIDIA Corporation)
• Inventors: Yang, Lin, Wheeler, Mark Damon
• Primary Examiner(s): Couso, Yon J
• Application Number:
  US15/857,417
• Publication Number:
  US 20180192059A1
• Time in Patent Office: 614 Days
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: G01C 11/025 by scanning the object
• CPC Class Codes: G01C 21/30 Map- or contour-matching
• CPC Class Codes: G01C 21/3841 Data obtained from two or more sources, e.g. probe vehicles
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G01C 21/3867 Geometry of map features, e.g. shape points, polygons or for simplified maps
• CPC Class Codes: G01C 21/3874 Structures specially adapted for data searching and retrieval
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/90 using synthetic aperture techniques
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: G06T 9/20 Contour coding, e.g. using detection of edges
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object

Abstract

Embodiments relate to methods for efficiently encoding sensor data captured by an autonomous vehicle and building a high definition map using the encoded sensor data. The sensor data can be LiDAR data which is expressed as multiple image representations. Image representations that include important LiDAR data undergo a lossless compression while image representations that include LiDAR data that is more error-tolerant undergo a lossy compression. Therefore, the compressed sensor data can be transmitted to an online system for building a high definition map. When building a high definition map, entities, such as road signs and road lines, are constructed such that when encoded and compressed, the high definition map consumes less storage space. The positions of entities are expressed in relation to a reference centerline in the high definition map. Therefore, each position of an entity can be expressed in fewer numerical digits in comparison to conventional methods.

Claims

1. A method comprising:

receiving, from a sensor mounted on a vehicle, sensor data describing locations within an environment around the vehicle captured by the sensor;

determining based on the received sensor data a plurality of sensor orientation datasets, each sensor orientation dataset comprising a pitch value, a yaw value, a range value, and an intensity value;

constructing one or more image representations from the plurality of sensor orientation datasets, comprising, for each image representation, repeating for a plurality of pixels;

identifying, in the image representation, a pixel location corresponding to a combination of a pitch value and a yaw value of a sensor orientation dataset, and

determining a value of the identified pixel from one of the range value or the intensity value of the sensor orientation dataset, and

setting the determined value for the pixel at the identified pixel location;

compressing each of the one or more image representations to generate compressed codes representing the plurality of sensor orientation datasets; and

transmitting the compressed codes for generation of a high definition map of the environment around the vehicle, the high definition map for use in driving by one or more autonomous vehicles.

2. The method of claim 1, wherein a number of rows of each of the one or more image representations is dependent on a number of lasers of the vehicle sensor.

3. The method of claim 1, wherein a number of columns of each of the one or more image representations is dependent on a minimum angular resolution of the vehicle sensor.

4. The method of claim 1, wherein constructing one or more image representations comprises:

generating a first image representation, wherein a value of a pixel of the first image representation is a most significant byte of the range value of the dataset;

generating a second image representation, wherein a value of a pixel of the second image representation is a least significant byte of the range value of the dataset; and

generating a third image representation, wherein a value of a pixel of the third image representation is a byte of the intensity value of the dataset.

5. The method of claim 4, wherein a step value of the most significant byte and least significant byte of the range value is dependent on a maximum range detectable by the vehicle sensor.

6. The method of claim 4, wherein constructing one or more image representations of the three dimensional data further comprises:

generating a fourth image representation, wherein a value of a pixel of the fourth image representation is a color value.

7. The method of claim 4, wherein encoding each of the one or more image representations comprises encoding the first image representation using a lossless encoding process.

8. The method of claim 4, wherein encoding each of the one or more image representations comprises encoding the second image representation and the third image representation using a lossy encoding process.

9. The method of claim 1, wherein constructing one or more image representations of the three dimensional data further comprises:

for each of the one or more image representations, identifying one or more pixels of the image representation with a value indicating missing data; and

for each of the one or more pixels with a non-existent value, modifying the pixel with the value indicating missing data to store a value based on other pixels of the image representation.

10. The method of claim 9, further comprising:

generating a bitmap indicating a presence of one or more pixels in the one or more image representations that has the value indicating missing data;

encoding the generated bitmap; and

transmitting the encoded bitmap, the encoded bitmap further used for generation of the high definition map of the environment around the vehicle.

11. A non-transitory computer readable storage medium comprising computer instructions that, when executed by a processor, cause the processor to:

receive, from a sensor mounted on a vehicle, sensor data describing locations within an environment around the vehicle captured by the sensor;

determining based on the received sensor data, a plurality of sensor orientation datasets, each sensor orientation dataset comprising a pitch value, a yaw value, a range value, and an intensity value;

construct one or more image representations from the plurality of sensor orientation datasets, comprising, for each image representation, repeating for a plurality of pixels;

identify, in the image representation, a pixel location corresponding to a combination of a pitch value and a yaw value of a sensor orientation dataset, and

determine a value of the identified pixel from one of the range value or the intensity value of the sensor orientation dataset, and

set the determined value for the pixel at the identified pixel location;

compress each of the one or more image representations to generate compressed codes representing the plurality of sensor orientation datasets; and

transmit the compressed codes for generation of a high definition map of the environment around the vehicle, the high definition map for use in driving by one or more autonomous vehicles.

12. The non-transitory computer storage of claim 11, wherein a number of rows of each of the one or more image representations is dependent on a number of lasers of the vehicle sensor.

13. The non-transitory computer storage of claim 11, wherein a number of columns of each of the one or more image representations is dependent on a minimum angular resolution of the vehicle sensor.

14. The non-transitory computer storage of claim 11, wherein the computer instructions that cause the processor to construct one or more image representations further comprise instructions that, when executed by the processor, cause the processor to:

generate a first image representation, wherein a value of a pixel of the first image representation is a most significant byte of the range value of the dataset;

generate a second image representation, wherein a value of a pixel of the second image representation is a least significant byte of the range value of the dataset; and

generate a third image representation, wherein a value of a pixel of the third image representation is a byte of the intensity value of the dataset.

15. The non-transitory computer storage of claim 14, wherein a step value of the most significant byte and least significant byte of the range value is dependent on a maximum range detectable by the vehicle sensor.

16. The non-transitory computer storage of claim 14, wherein the computer instructions that cause the processor to encode each of the one or more image representations further comprise computer instructions that, when executed by the processor, cause the processor to encode the first image representation using a lossless encoding process.

17. The non-transitory computer storage of claim 14, wherein the computer instructions that cause the processor to encode each of the one or more image representations further comprise computer instructions that, when executed by the processor, cause the processor to encode the second image representation and the third image representation using a lossy encoding process.

18. The non-transitory computer storage of claim 11, wherein the computer instructions that cause the processor to construct one or more image representations of the three dimensional data further comprises instructions that, when executed by the processor, cause the processor to:

for each of the one or more image representations, identify one or more pixels of the image representation with a value indicating missing data; and

for each of the one or more pixels with a non-existent value, modify the pixel with the value indicating missing data to store a value based on other pixels of the image representation.

19. The non-transitory computer storage of claim 18, further comprising computer instructions that, when executed by the processor, cause the processor to:

generate a bitmap indicating a presence of one or more pixels in the one or more image representations that has the value indicating missing data;

encode the generated bitmap; and

transmit the encoded bitmap, the encoded bitmap further used for generation of the high definition map of the environment around the vehicle.

20. A computer-system comprising:

a processor; and

a non-transitory computer readable storage medium comprising computer instructions that, when executed by a processor, cause the processor to;

receive, from a sensor mounted on a vehicle, sensor data describing locations within an environment around the vehicle captured by the sensor;

determine based on the received sensor data, a plurality of sensor orientation datasets, each sensor orientation dataset comprising a pitch value, a yaw value, a range value, and an intensity value;

construct one or more image representations from the plurality of sensor orientation datasets, comprising, for each image representation, repeating for a plurality of pixels;

identify, in the image representation, a pixel location corresponding to a combination of a pitch value and a yaw value of a sensor orientation dataset, and

determine a value of the identified pixel from one of the range value or the intensity value of the sensor orientation dataset, and

set the determined value for the pixel at the identified pixel location;

compress each of the one or more image representations to generate compressed codes representing the plurality of sensor orientation datasets; and

transmit the compressed codes for generation of a high definition map of the environment around the vehicle, the high definition map for use in driving by one or more autonomous vehicles.

21. The computer-system of claim 20, wherein a number of rows of each of the one or more image representations is dependent on a number of lasers of the vehicle sensor.

22. The computer-system of claim 20, wherein a number of columns of each of the one or more image representations is dependent on a minimum angular resolution of the vehicle sensor.

23. The computer-system of claim 20, wherein the computer instructions that cause the processor to construct one or more image representations further comprise instructions that, when executed by the processor, cause the processor to:

generate a first image representation, wherein a value of a pixel of the first image representation is a most significant byte of the range value of the dataset;

generate a second image representation, wherein a value of a pixel of the second image representation is a least significant byte of the range value of the dataset; and

generate a third image representation, wherein a value of a pixel of the third image representation is a byte of the intensity value of the dataset.

24. The computer-system of claim 23, wherein a step value of the most significant byte and least significant byte of the range value is dependent on a maximum range detectable by the vehicle sensor.

25. The computer-system of claim 23, wherein the computer instructions that cause the processor to encode each of the one or more image representations further comprise computer instructions that, when executed by the processor, cause the processor to encode the first image representation using a lossless encoding process.

26. The computer-system of claim 23, wherein the computer instructions that cause the processor to encode each of the one or more image representations further comprise computer instructions that, when executed by the processor, cause the processor to encode the second image representation and the third image representation using a lossy encoding process.

27. The computer-system of claim 20, wherein the computer instructions that cause the processor to construct one or more image representations of the three dimensional data further comprise instructions that, when executed by the processor, cause the processor to:

for each of the one or more image representations, identify one or more pixels of the image representation with a value indicating missing data; and

for each of the one or more pixels with a non-existent value, modify the pixel with the value indicating missing data to store a value based on other pixels of the image representation.

28. The computer-system of claim 27, further comprising computer instructions that, when executed by the processor, cause the processor to:

generate a bitmap indicating a presence of one or more pixels in the one or more image representations that has the value indicating missing data;

encode the generated bitmap; and

transmit the encoded bitmap, the encoded bitmap further used for generation of the high definition map of the environment around the vehicle.