PROCESSING POINT CLOUDS OF VEHICLE SENSORS HAVING VARIABLE SCAN LINE DISTRIBUTIONS USING INTERPOLATION FUNCTIONS

• US 20190180502A1
• Filed: 10/31/2018
• Published: 06/13/2019
• Est. Priority Date: 12/13/2017
• Status: Active Grant

• Current Assignee: Luminar Technologies, Inc.
• Original Assignee: Luminar Technologies, Inc.
• Inventors: Englard, Benjamin, Danziger, Eric C., Russell, Austin K.
• Granted Patent: US 10,768,304 B2
• Time in Patent Office: Days
• CPC Class Codes: B60W 10/04 including control of propulsion units
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2552/15 Road slope, i.e. the inclination of a road segment in the longitudinal direction
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2555/20 Ambient conditions, e.g. wind or rain
• CPC Class Codes: B60W 2555/40 Altitude
• CPC Class Codes: B60W 2555/60 Traffic rules, e.g. speed limits or right of way
• CPC Class Codes: B60W 2720/106 Longitudinal acceleration
• CPC Class Codes: B60W 2720/12 Lateral speed
• CPC Class Codes: B60W 2720/125 Lateral acceleration
• CPC Class Codes: B60W 30/18145 Cornering
• CPC Class Codes: B60W 30/18163 Lane change; Overtaking manoeuvres
• CPC Class Codes: B60W 50/06 Improving the dynamic response of the control system, e.g. improving the speed of regulation or avoiding hunting or overshoot
• CPC Class Codes: G01S 13/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S13/46
• CPC Class Codes: G01S 13/426 Scanning radar, e.g. 3D radar G01S13/66 takes precedence
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/89 for mapping or imaging
• CPC Class Codes: G01S 13/931 : of land vehicles
• CPC Class Codes: G01S 17/66 : Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/86 : Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 : of land vehicles
• CPC Class Codes: G01S 2013/9323 : Alternative operation using light waves
• CPC Class Codes: G01S 2013/9324 : Alternative operation using ultrasonic waves
• CPC Class Codes: G01S 7/40 : Means for monitoring or calibrating
• CPC Class Codes: G01S 7/4026 : Antenna boresight
• CPC Class Codes: G01S 7/403 : in azimuth, i.e. in the horizontal plane
• CPC Class Codes: G01S 7/4034 : in elevation, i.e. in the vertical plane
• CPC Class Codes: G01S 7/417 : involving the use of neural networks
• CPC Class Codes: G01S 7/4808 : Evaluating distance, position or velocity data
• CPC Class Codes: G01S 7/4817 : relating to scanning
• CPC Class Codes: G01S 7/4972 : Alignment of sensor
• CPC Class Codes: G05D 1/0088 : characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0094 : involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/0221 : involving a learning process
• CPC Class Codes: G05D 1/0231 : using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0248 : in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0257 : using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G06F 18/214 : Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2163 : Partitioning the feature space
• CPC Class Codes: G06F 18/217 : Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/24 : Classification techniques
• CPC Class Codes: G06F 18/24143 : Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06N 3/045 : Combinations of networks
• CPC Class Codes: G06N 3/08 : Learning methods
• CPC Class Codes: G06N 3/086 : using evolutionary algorithms, e.g. genetic algorithms or genetic programming
• CPC Class Codes: G06N 5/01 : Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06T 15/08 : Volume rendering
• CPC Class Codes: G06T 17/05 : Geographic models
• CPC Class Codes: G06T 17/20 : Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2207/10028 : Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 : Training; Learning
• CPC Class Codes: G06T 2207/20084 : Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30252 : Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 3/4007 : based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/50 : Depth or shape recovery
• CPC Class Codes: G06V 10/764 : using classification, e.g. of video objects
• CPC Class Codes: G06V 10/776 : Validation; Performance evaluation
• CPC Class Codes: G06V 10/82 : using neural networks
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/59 : inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions

Abstract

A method for processing point clouds having variable spatial distributions of scan lines includes receiving a point cloud frame generated by a sensor configured to sense an environment through which a vehicle is moving. The point cloud frame includes scan lines arranged according to a particular spatial distribution. The method also includes either generating an enhanced point cloud frame with a larger number of points than the received point cloud frame, or constructing, by one or more processors and based on points of the received point cloud frame, a three-dimensional mesh. The method also includes generating, by performing an interpolation function on the enhanced point cloud frame or a virtual surface provided by the three-dimensional mesh, a normalized point cloud frame, and generating, using the normalized point cloud frame, signals descriptive of a current state of the environment through which the vehicle is moving.

Claims

1. A method for processing point clouds having variable spatial distributions of scan lines, the method comprising:

receiving a point cloud frame generated by a sensor configured to sense an environment through which a vehicle is moving, the point cloud frame including a plurality of scan lines arranged according to a particular spatial distribution;

either (i) generating, by one or more processors, an enhanced point cloud frame with a larger number of points than the received point cloud frame, or (ii) constructing, by one or more processors and based on points of the received point cloud frame, a three-dimensional mesh;

generating, by one or more processors performing an interpolation function on (i) the enhanced point cloud frame or (ii) a virtual surface provided by the three-dimensional mesh, a normalized point cloud frame; and

generating, by one or more processors and using the normalized point cloud frame, signals descriptive of a current state of the environment through which the vehicle is moving.

2. The method of claim 1, wherein the enhanced point cloud frame is generated using camera image data.

3. The method of claim 1, wherein the method comprises:

receiving camera image data generated by a camera disposed on or in the vehicle;

generating, using the camera image data, the enhanced point cloud frame; and

generating the normalized point cloud frame by performing the interpolation function on the enhanced point cloud frame.

4. The method of claim 3, wherein generating, using the camera image data, the enhanced point cloud frame includes:

transforming the received camera image data to points in a three-dimensional space; and

generating the enhanced point cloud using the received point cloud and the points in the three-dimensional space.

5. The method of claim 3, wherein performing the interpolation function on the enhanced point cloud frame includes sampling points of the enhanced point cloud in a manner that at least approximates a reference scan line distribution.

6. The method of claim 3, wherein the sensor is a lidar device.

7. The method of claim 1, wherein the particular spatial distribution of scan lines comprises a uniform distribution, a Gaussian distribution, a multimodal distribution, and/or an arbitrary distribution.

8. The method of claim 1, wherein the method comprises:

constructing, based on the points of the received point cloud frame, the three-dimensional mesh; and

generating the normalized point cloud frame by performing the interpolation function on the virtual surface provided by the three-dimensional mesh.

9. The method of claim 8, wherein:

constructing the three-dimensional mesh includes forming triangles from subsets of the points of the received point cloud frame; and

performing the interpolation function on the virtual surface provided by the three-dimensional mesh includes generating points within planes of the triangles and at positions that at least approximate a reference scan line distribution.

10. The method of claim 8, wherein generating the signals descriptive of the current state of the environment through which the vehicle is moving includes:

partitioning the normalized point cloud frame into a plurality of portions in accordance with probable boundaries between separate physical objects; and

classifying at least some of the plurality of portions according to object types.

11. A non-transitory computer-readable medium storing thereon instructions executable by one or more processors to implement a self-driving control architecture of a vehicle, the self-driving control architecture comprising:

a perception component configured to

receive a point cloud frame generated by a sensor configured to sense an environment through which a vehicle is moving, the point cloud frame including a plurality of scan lines arranged according to a particular spatial distribution,

either (i) generate an enhanced point cloud frame with a larger number of points than the received point cloud frame, or (ii) construct, based on points of the received point cloud frame, a three-dimensional mesh,

generate, by performing an interpolation function on (i) the enhanced point cloud frame or (ii) a virtual surface provided by the three-dimensional mesh, a normalized point cloud frame, and

generate, using the normalized point cloud frame, signals descriptive of a current state of the environment through which the vehicle is moving.

12. The non-transitory computer-readable medium of claim 11, wherein the perception component is configured to:

receive camera image data generated by a camera disposed on or in the vehicle;

generate, using the camera image data, the enhanced point cloud frame; and

generate the normalized point cloud frame by performing the interpolation function on the enhanced point cloud frame.

13. The non-transitory computer-readable medium of claim 12, wherein the perception component is configured to generate, using the camera image data, the enhanced point cloud frame at least by:

transforming the received camera image data to points in a three-dimensional space; and

generating the enhanced point cloud using the received point cloud and the points in the three-dimensional space.

14. The non-transitory computer-readable medium of claim 12, wherein the perception component is configured to perform the interpolation function on the enhanced point cloud frame at least by sampling points of the enhanced point cloud in a manner that at least approximates a reference scan line distribution.

15. The non-transitory computer-readable medium of claim 12, wherein the sensor is a lidar device.

16. The non-transitory computer-readable medium of claim 11, wherein the perception component is configured to:

construct, based on the points of the received point cloud frame, the three-dimensional mesh; and

generate the normalized point cloud frame by performing the interpolation function on the virtual surface provided by the three-dimensional mesh.

17. The non-transitory computer-readable medium of claim 16, wherein the perception component is configured to:

construct the three-dimensional mesh at least by forming triangles from subsets of the points of the received point cloud frame; and

perform the interpolation function on the virtual surface provided by the three-dimensional mesh at least by generating points within planes of the triangles and at positions that at least approximate a reference scan line distribution.

18. The non-transitory computer-readable medium of claim 16, wherein the perception component is configured to generate the signals descriptive of the current state of the environment through which the vehicle is moving at least by:

partitioning the normalized point cloud frame into a plurality of portions in accordance with probable boundaries between separate physical objects; and

classifying at least some of the plurality of portions according to object types.

19. A vehicle comprising:

a sensor configured to sense an environment through which the vehicle is moving and generate point cloud frames;

one or more operational subsystems; and

a computing system configured to

receive a point cloud frame generated by the sensor, the point cloud frame including a plurality of scan lines arranged according to a particular spatial distribution,

either (i) generate an enhanced point cloud frame with a larger number of points than the received point cloud frame, or (ii) construct, based on points of the received point cloud frame, a three-dimensional mesh,

generate, by performing an interpolation function on (i) the enhanced point cloud frame or (ii) a virtual surface provided by the three-dimensional mesh, a normalized point cloud frame,

generate, using the normalized point cloud frame, signals descriptive of a current state of the environment through which the vehicle is moving,

generate driving decisions based on the signals descriptive of the current state of the environment, and

cause the one or more operational subsystems of the vehicle to maneuver the vehicle in accordance with the generated driving decisions.

20. The vehicle of claim 19, wherein the vehicle further comprises a camera disposed on or in the vehicle, and wherein the computing system is configured to:

receive camera image data generated by the camera;

generate, using the camera image data, the enhanced point cloud frame; and

generate the normalized point cloud frame by performing the interpolation function on the enhanced point cloud frame.

21. The vehicle of claim 20, wherein the computing system is configured to generate, using the camera image data, the enhanced point cloud frame at least by:

transforming the received camera image data to points in a three-dimensional space; and

generating the enhanced point cloud using the received point cloud and the points in the three-dimensional space.

22. The method of claim 19, wherein the computing system is configured to:

construct, based on the points of the received point cloud frame, the three-dimensional mesh; and

generate the normalized point cloud frame by performing the interpolation function on the virtual surface provided by the three-dimensional mesh.