Controlling vehicle sensors based on dynamic objects

• US 10,627,521 B2
• Filed: 10/31/2018
• Issued: 04/21/2020
• Est. Priority Date: 12/13/2017
• Status: Active Grant

• Current Assignee: Luminar Technologies, Inc.
• Original Assignee: Luminar Technologies, Inc.
• Inventors: Englard, Benjamin, Danziger, Eric C., Russell, Austin K.
• Primary Examiner(s): Abraham, Samantha K
• Application Number:
  US16/176,473
• Publication Number:
  US 20190179023A1
• Time in Patent Office: 538 Days
• CPC Class Codes: B60W 10/04 including control of propulsion units
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2552/15 Road slope, i.e. the inclination of a road segment in the longitudinal direction
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2555/20 Ambient conditions, e.g. wind or rain
• CPC Class Codes: B60W 2555/40 Altitude
• CPC Class Codes: B60W 2555/60 Traffic rules, e.g. speed limits or right of way
• CPC Class Codes: B60W 2720/106 Longitudinal acceleration
• CPC Class Codes: B60W 2720/12 Lateral speed
• CPC Class Codes: B60W 2720/125 Lateral acceleration
• CPC Class Codes: B60W 30/18145 Cornering
• CPC Class Codes: B60W 30/18163 Lane change; Overtaking manoeuvres
• CPC Class Codes: B60W 50/06 Improving the dynamic response of the control system, e.g. improving the speed of regulation or avoiding hunting or overshoot
• CPC Class Codes: G01S 13/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S13/46
• CPC Class Codes: G01S 13/426 Scanning radar, e.g. 3D radar G01S13/66 takes precedence
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/89 for mapping or imaging
• CPC Class Codes: G01S 13/931 : of land vehicles
• CPC Class Codes: G01S 17/66 : Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/86 : Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 : of land vehicles
• CPC Class Codes: G01S 2013/9323 : Alternative operation using light waves
• CPC Class Codes: G01S 2013/9324 : Alternative operation using ultrasonic waves
• CPC Class Codes: G01S 7/40 : Means for monitoring or calibrating
• CPC Class Codes: G01S 7/4026 : Antenna boresight
• CPC Class Codes: G01S 7/403 : in azimuth, i.e. in the horizontal plane
• CPC Class Codes: G01S 7/4034 : in elevation, i.e. in the vertical plane
• CPC Class Codes: G01S 7/417 : involving the use of neural networks
• CPC Class Codes: G01S 7/4808 : Evaluating distance, position or velocity data
• CPC Class Codes: G01S 7/4817 : relating to scanning
• CPC Class Codes: G01S 7/4972 : Alignment of sensor
• CPC Class Codes: G05D 1/0088 : characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0094 : involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/0221 : involving a learning process
• CPC Class Codes: G05D 1/0231 : using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0248 : in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0257 : using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G06F 18/214 : Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2163 : Partitioning the feature space
• CPC Class Codes: G06F 18/217 : Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/24 : Classification techniques
• CPC Class Codes: G06F 18/24143 : Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06N 3/045 : Combinations of networks
• CPC Class Codes: G06N 3/08 : Learning methods
• CPC Class Codes: G06N 3/086 : using evolutionary algorithms, e.g. genetic algorithms or genetic programming
• CPC Class Codes: G06N 5/01 : Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06T 15/08 : Volume rendering
• CPC Class Codes: G06T 17/05 : Geographic models
• CPC Class Codes: G06T 17/20 : Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2207/10028 : Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 : Training; Learning
• CPC Class Codes: G06T 2207/20084 : Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30252 : Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 3/4007 : based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/50 : Depth or shape recovery
• CPC Class Codes: G06V 10/764 : using classification, e.g. of video objects
• CPC Class Codes: G06V 10/776 : Validation; Performance evaluation
• CPC Class Codes: G06V 10/82 : using neural networks
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/59 : inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions

Abstract

A method for controlling at least a first vehicle sensor includes receiving sensor data generated by one or more vehicle sensors that are configured to sense an environment through which the vehicle is moving, and identifying, based on the received sensor data, one or more current and/or predicted positions of one or more dynamic objects that are currently moving, or are capable of movement, within the environment. The method also includes causing, based on the current and/or predicted positions of the dynamic objects, an area of focus of the first sensor to be adjusted, at least by causing (i) a field of regard of the first sensor, and/or (ii) a spatial distribution of scan lines produced by the first sensor, to be adjusted.

Claims

1. A method for controlling a lidar device of a vehicle, the method comprising:

receiving sensor data generated by one or more sensors of the vehicle, wherein the one or more sensors are configured to sense an environment through which the vehicle is moving;

identifying, by one or more processors and based on the received sensor data, one or more current and one or more predicted positions of one or more dynamic objects that are currently moving, or are capable of movement, within the environment, wherein identifying a predicted position of at least one of the dynamic objects is based at least in part on applying, to the at least one dynamic object, one or more rules of the road on which the at least one dynamic object is operating; and

causing, by one or more processors and based on the current and predicted positions of the dynamic objects, an area of focus of the lidar device to be adjusted, wherein causing the area of focus to be adjusted includes causing a spatial distribution of scan lines produced by the lidar device to be adjusted, comprising increasing a density of the scan lines in one or more regions associated with the current and/or predicted positions of the dynamic objects.

2. The method of claim 1, wherein identifying the one or more current and the one or more predicted positions of the one or more dynamic objects includes:

generating, based on the received sensor data, signals descriptive of a current state of the environment; and

identifying, based on the signals descriptive of the current state of the environment, the one or more current and the one or more predicted positions of the one or more dynamic objects.

3. The method of claim 2, wherein generating the signals descriptive of the current state of the environment includes:

partitioning a digital representation of the environment into a plurality of portions in accordance with probable boundaries between separate physical objects, the digital representation including, or being derived from, the received sensor data;

classifying at least some of the plurality of portions according to object types; and

tracking movement of classified objects over time.

4. The method of claim 3, wherein identifying the one or more current and the one or more predicted positions of the one or more dynamic objects includes:

determining that the at least one dynamic object has been classified as an object type that is known to be capable of movement; and

identifying a current position and a predicted position of the at least one dynamic object.

5. The method of claim 2, wherein identifying, based on the signals descriptive of the current state of the environment, the one or more current and the one or more predicted positions of the one or more dynamic objects includes:

generating, based on the signals descriptive of the current state of the environment, signals descriptive of one or more future states of the environment; and

identifying, based on the signals descriptive of the one or more future states of the environment, the one or more predicted positions of the one or more dynamic objects.

6. The method of claim 1, wherein identifying the predicted position of the at least one dynamic object is further based at least in part on a speed and direction of the at least one dynamic object.

7. The method of claim 1, wherein causing the area of focus of the lidar device to be adjusted further includes causing a field of regard of the lidar device to be adjusted.

8. The method of claim 7, wherein causing the field of regard of the lidar device to be adjusted includes causing a vertical width and/or a horizontal width of the field of regard to be adjusted.

9. The method of claim 7, wherein causing the field of regard of the lidar device to be adjusted includes centering the field of regard on one of the predicted positions where one or more of the dynamic objects will be located at a time in a near future.

10. The method of claim 7, wherein:

identifying the one or more current and the one or more predicted positions of the one or more dynamic objects includes identifying an area having a high density of dynamic objects; and

causing the field of regard of the lidar device to be adjusted includes causing the field of regard to be centered and/or sized based on the identified area having a high density of dynamic objects.

11. The method of claim 7, wherein:

identifying the one or more current and the one or more predicted positions of the one or more dynamic objects includes identifying current and predicted positions of a plurality of dynamic objects; and

causing the field of regard of the lidar device to be adjusted includes preventing a vertical width and/or a horizontal width of the field of regard from being reduced to a level that excludes any of the plurality of dynamic objects.

12. The method of claim 1, wherein the one or more sensors include only the lidar device.

13. The method of claim 1, wherein the one or more sensors include a camera.

14. The method of claim 1, wherein the one or more sensors include the lidar device, and wherein the sensor data includes a point cloud generated by the lidar device.

15. The method of claim 1, wherein one or both of (i) identifying the one or more current and the one or more predicted positions of the one or more dynamic objects, and (ii) causing, based on the one or more current and predicted positions of the one or more dynamic objects, the area of focus of the lidar device to be adjusted, are performed using a machine learning based attention model.

16. The method of claim 15, further comprising training, by one or more processors, the attention model.

17. The method of claim 16, wherein training the attention model includes training the attention model using reinforcement learning.

18. The method of claim 17, wherein training the attention model using reinforcement learning includes providing rewards based on achievement of a number of driving goals.

19. The method of claim 16, wherein training the attention model includes training the attention model using an evolutionary algorithm.

20. The method of claim 16, wherein training the attention model includes:

receiving data indicating visual focus of a human driver during a real or simulated driving trip;

using the received data indicating visual focus of the human driver to generate labels for a labeled data set; and

performing supervised training of the attention model using the labeled data set.

21. The method of claim 1, wherein increasing the density of the scan lines comprises arranging the scan lines based on a continuous mathematical distribution, wherein a peak of the mathematical distribution corresponds to a region of the one or more regions.

22. A non-transitory computer-readable medium storing thereon instructions executable by one or more processors to implement a sensor control architecture for controlling a lidar device of a vehicle, the sensor control architecture being configured to:

receive sensor data generated by one or more sensors of the vehicle, wherein the one or more sensors are configured to sense an environment through which the vehicle is moving;

identify, based on the received sensor data, one or more current and one or more predicted positions of one or more dynamic objects that are currently moving, or are capable of movement, within the environment, wherein identifying a predicted position of at least one of the dynamic objects is based at least in part on applying, to the at least one dynamic object, one or more rules of the road on which the at least one dynamic object is operating; and

cause, based on the current and predicted positions of the dynamic objects, an area of focus of the lidar device to be adjusted, at least by causing a spatial distribution of scan lines produced by the lidar device to be adjusted, comprising increasing a density of the scan lines in one or more regions associated with the current and/or predicted positions of the dynamic objects.

23. The non-transitory computer-readable medium of claim 22, wherein the sensor control architecture is configured to identify the one or more current and the one or more predicted positions of the one or more dynamic objects at least by:

generating, based on the received sensor data, signals descriptive of a current state of the environment; and

identifying, based on the signals descriptive of the current state of the environment, the one or more current and the one or more predicted positions of the one or more dynamic objects.

24. The non-transitory computer-readable medium of claim 23, wherein the sensor control architecture is configured to generate the signals descriptive of the current state of the environment at least by:

partitioning a digital representation of the environment into a plurality of portions in accordance with probable boundaries between separate physical objects, the digital representation including, or being derived from, the received sensor data;

classifying at least some of the plurality of portions according to object types; and

tracking movement of classified objects over time.

25. The non-transitory computer-readable medium of claim 23, wherein the sensor control architecture is configured to identify the one or more current and the one or more predicted positions of the one or more dynamic objects at least by:

determining that the at least one dynamic object has been classified as an object type that is known to be capable of movement; and

identifying a current position and a predicted position of the at least one dynamic object.

26. The non-transitory computer-readable medium of claim 22, wherein identify the predicted position of the at least one dynamic object is further based at least in part on a speed and direction of the at least one dynamic object.

27. The non-transitory computer-readable medium of claim 22, wherein the sensor control architecture is further configured to cause, based on the current and/or predicted positions of the dynamic objects, a field of regard of the lidar device to be adjusted.

28. The non-transitory computer-readable medium of claim 22, wherein the one or more sensors include a camera.

29. The non-transitory computer-readable medium of claim 22, wherein the sensor control architecture includes a machine learning based attention model.

30. The non-transitory computer-readable medium of claim 29, wherein the attention model is trained using (i) reinforcement learning, (ii) evolutionary algorithms, or (iii) supervised learning with labels corresponding to focus of a human driver.