Generating data using radar observation model based on machine learning

• US 10,732,261 B1
• Filed: 12/31/2019
• Issued: 08/04/2020
• Est. Priority Date: 12/31/2019
• Status: Active Grant

• Current Assignee: Aurora Operations Incorporated
• Original Assignee: Aurora Innovation, Inc.
• Inventors: Wang, Shaogang, Eade, Ethan, Smith, Warren
• Primary Examiner(s): Gregory, Bernarr E
• Application Number:
  US16/732,152
• Time in Patent Office: 217 Days
• Field of Search: None
• CPC Class Codes: G01S 13/44 Monopulse radar, i.e. simultaneous lobing
• CPC Class Codes: G01S 13/584 adapted for simultaneous range and velocity measurements
• CPC Class Codes: G01S 13/726 Multiple target tracking
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/878 Combination of several spaced transmitters or receivers of known location for determining the position of a transponder or a reflector G01S13/874 takes precedence
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 2013/9323 Alternative operation using light waves
• CPC Class Codes: G01S 2013/9324 Alternative operation using ultrasonic waves
• CPC Class Codes: G01S 2013/93271 in the front of the vehicles
• CPC Class Codes: G01S 2013/93272 in the back of the vehicles
• CPC Class Codes: G01S 2013/93274 on the side of the vehicles
• CPC Class Codes: G01S 7/4026 Antenna boresight
• CPC Class Codes: G01S 7/403 in azimuth, i.e. in the horizontal plane
• CPC Class Codes: G01S 7/417 involving the use of neural networks
• CPC Class Codes: G05D 1/0221 involving a learning process
• CPC Class Codes: G05D 1/0257 using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06N 20/10 : using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/044 : Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 : Combinations of networks
• CPC Class Codes: G06N 3/084 : Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 5/01 : Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 7/01 : Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/10024 : Color image
• CPC Class Codes: G06T 2207/10028 : Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 : Training; Learning
• CPC Class Codes: G06T 2207/20084 : Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30241 : Trajectory
• CPC Class Codes: G06T 2207/30261 : Obstacle
• CPC Class Codes: G06T 2210/12 : Bounding box
• CPC Class Codes: G06T 7/207 : for motion estimation over a hierarchy of resolutions multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53
• CPC Class Codes: G06T 7/246 : using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/277 : involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/74 : involving reference images or patches

Abstract

A method includes obtaining a first track associated with a first time. A first track associated with a first time is obtained. First predicted state data associated with a second time that is later than the first time, are generated based on the first track. Radar measurement data associated with the second time are obtained from one or more radar sensors. Track data are generated by a machine learning model based on the first predicted state data and the radar measurement data. Second predicted state data associated with the second time are generated based on the first track. A second track associated with the second time is generated based on the track data and the second predicted state data. The second track associated with the second time is provided to an autonomous vehicle control system for autonomous control of a vehicle.

Claims

1. A method comprising:

obtaining a first track associated with a first time;

based on the first track, generating first predicted state data associated with a second time that is later than the first time;

obtaining, from one or more radar sensors, radar measurement data associated with the second time;

based on the first predicted state data and the radar measurement data, generating track data by a machine learning model;

based on the first track, generating second predicted state data associated with the second time;

based on the track data and the second predicted state data, generating a second track associated with the second time; and

providing the second track associated with the second time to an autonomous vehicle control system for autonomous control of a vehicle.

2. The method of claim 1, wherein each of the first predicted state data and the second predicted state data comprises predicted position information of the first track associated with the second time.

3. The method of claim 1, wherein each of generating the first predicted state data associated with the second time and generating the second predicted state data associated with the second time comprises:

generating predicted position information of the first track associated with the second time by using a motion model based on a time difference between the first time and the second time.

4. The method of claim 1, wherein the radar measurement data represents a plurality of radar points measured by the one or more radar sensors.

5. The method of claim 1, wherein the radar measurement data represents a plurality of radar points associated with a track, each radar point being associated with at least one or more values of range, azimuth, range rate, or intensity.

6. The method of claim 1, wherein generating the track data by the machine learning model comprises:

generating a predicted bounding box of the first track based on the first predicted state data;

generating associated data by sampling radar points from the radar measurement data and associating the sampled radar points with a side of the predicted bounding box of the first track;

encoding the sampled radar points into feature vectors; and

generating the track data as output data of the machine learning model which has the feature vectors as input data thereof.

7. The method of claim 6, wherein the feature vectors comprises one or more input channels, each input channel comprising a plurality of azimuth angle bins representing a field of view of a plurality of radar points.

8. The method of claim 7, wherein the plurality of azimuth angle bins of each input channel contain values representing one of maximum range, minimum range, maximum range rate, minimum range rate, or maximum intensity.

9. The method of claim 1, further comprising:

generating a proposal that represents a third track associated with the second time, which is different from the first track;

based on the proposal, generating a combined track of at least the second track and the third track; and

providing the combined track associated with the second time to the autonomous vehicle control system for autonomous control of the vehicle.

10. The method of claim 9, wherein the proposal is generated without using a machine learning model.

11. The method of claim 9, wherein the proposal is generated based on the radar measurement data.

12. The method of claim 1, wherein the track data includes data representing a single radar point associated with a center of a track.

13. The method of claim 12, wherein the track data comprises one or more output channels, each output channel comprising a plurality of azimuth angle bins representing a field of view of the single radar point.

14. The method of claim 13, wherein the plurality of azimuth angle bins of each output channel contain values representing one of weight, range, or range rate.

15. A system comprising one or more processors and one or more memories operably coupled with the one or more processors, wherein the one or more memories store instructions that, in response to the execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:

obtaining a first track associated with a first time;

based on the first track, generating first predicted state data associated with a second time that is later than the first time;

obtaining, from one or more radar sensors, radar measurement data associated with the second time;

based on the first predicted state data and the radar measurement data, generating track data by a machine learning model;

based on the first track, generating second predicted state data associated with the second time;

based on the track data and the second predicted state data, generating a second track associated with the second time; and

providing the second track associated with the second time to an autonomous vehicle control system for autonomous control of a vehicle.

16. The system of claim 15, wherein generating the track data by the machine learning model comprises:

generating a predicted bounding box of the first track based on the first predicted state data;

generating associated data by sampling radar points from the radar measurement data and associating the sampled radar points with a side of the predicted bounding box of the first track;

encoding the sampled radar points into feature vectors; and

generating the track data as output data of the machine learning model which has the feature vectors as input data thereof.

17. The system of claim 16, wherein the instructions, in response to the execution of the instructions by one or more processors, cause the one or more processors to perform:

generating a proposal that represents a third track associated with the second time, which is different from the first track;

based on the proposal, generating a combined track of at least the second track and the third track; and

providing the combined track associated with the second time to the autonomous vehicle control system for autonomous control of the vehicle.

18. At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause one or more processors to perform the following operations:

obtaining a first track associated with a first time;

based on the first track, generating first predicted state data associated with a second time that is later than the first time;

obtaining, from one or more radar sensors, radar measurement data associated with the second time;

based on the first predicted state data and the radar measurement data, generating track data by a machine learning model;

based on the first track, generating second predicted state data associated with the second time;

based on the track data and the second predicted state data, generating a second track associated with the second time; and

providing the second track associated with the second time to an autonomous vehicle control system for autonomous control of a vehicle.

19. The at least one non-transitory computer-readable medium of claim 18, wherein generating the track data by the machine learning model comprises:

generating a predicted bounding box of the first track based on the first predicted state data;

generating associated data by sampling radar points from the radar measurement data and associating the sampled radar points with a side of the predicted bounding box of the first track;

encoding the sampled radar points into feature vectors; and

generating the track data as output data of the machine learning model which has the feature vectors as input data thereof.

20. The at least one non-transitory computer-readable medium of claim 18, wherein the instructions, in response to the execution of the instructions by one or more processors, cause the one or more processors to perform:

generating a proposal that represents a third track associated with the second time, which is different from the first track;

based on the proposal, generating a combined track of at least the second track and the third track; and

providing the combined track associated with the second time to the autonomous vehicle control system for autonomous control of the vehicle.