Training machine learning model based on training instances with: training instance input based on autonomous vehicle sensor data, and training instance output based on additional vehicle sensor data

• US 10,676,085 B2
• Filed: 10/29/2018
• Issued: 06/09/2020
• Est. Priority Date: 04/11/2018
• Status: Active Grant

• Current Assignee: Aurora Operations Incorporated
• Original Assignee: Aurora Innovation, Inc.
• Inventors: Smith, Warren, Eade, Ethan, Anderson, Sterling J., Bagnell, James Andrew, Nabbe, Bartholomeus C., Urmson, Christopher Paul
• Primary Examiner(s): Yang, Qian
• Application Number:
  US16/173,669
• Publication Number:
  US 20190318206A1
• Time in Patent Office: 589 Days
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2554/4041 Position
• CPC Class Codes: B60W 2554/803 Relative lateral speed
• CPC Class Codes: B60W 2754/00 Output or target parameters relating to objects
• CPC Class Codes: B60W 30/0956 the prediction being responsive to traffic or environmental parameters
• CPC Class Codes: B60W 40/04 Traffic conditions
• CPC Class Codes: G01S 17/34 using transmission of continuous, frequency-modulated waves while heterodyning the received signal, or a signal derived therefrom, with a locally-generated signal related to the contemporaneously transmitted signal
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/58 Velocity or trajectory determination systems; Sense-of-movement determination systems
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2178 based on feedback of a supervisor
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

Various implementations described herein generate training instances that each include corresponding training instance input that is based on corresponding sensor data of a corresponding autonomous vehicle, and that include corresponding training instance output that is based on corresponding sensor data of a corresponding additional vehicle, where the corresponding additional vehicle is captured at least in part by the corresponding sensor data of the corresponding autonomous vehicle. Various implementations train a machine learning model based on such training instances. Once trained, the machine learning model can enable processing, using the machine learning model, of sensor data from a given autonomous vehicle to predict one or more properties of a given additional vehicle that is captured at least in part by the sensor data.

Claims

1. A method of training a machine learning model to be used in autonomous control of at least one autonomous vehicle, the method comprising:

generating a plurality of training instances, wherein generating each of the training instances includes;

generating training instance input of the training instance based on a corresponding instance of vision data from a vision component of a corresponding autonomous vehicle; and

generating a supervised training instance output of the training instance based on a corresponding instance of additional vehicle data, wherein the corresponding instance of additional vehicle data is based on one or more sensors of a corresponding additional vehicle that is captured by the corresponding instance of vision data, and wherein the corresponding instance of additional vehicle data indicates a corresponding current state of at least one dynamic property of the corresponding additional vehicle;

wherein the corresponding instance of additional vehicle data is used in generating the supervised training instance output based on determining that the corresponding instance of additional vehicle data temporally corresponds to the corresponding instance of vision data;

training the machine learning model based on the plurality of training instances; and

providing the trained machine learning model for use in control of the at least one autonomous vehicle.

2. The method of claim 1, wherein the corresponding instance of vision data is light Detection and Ranging (LIDAR) data and the vision component is a LIDAR component.

3. The method of claim 2, wherein the LIDAR component is a phase coherent LIDAR component and the LIDAR data comprises a group of LIDAR data points of a sensing cycle of the phase coherent LIDAR component, each of the LIDAR data points of the group indicating a corresponding range and a corresponding velocity for a corresponding point in the environment, and each being generated based on a corresponding sensing event of the sensing cycle.

4. The method of claim 1, wherein the supervised training instance output of the training instance further includes an indication of a bounding area that bounds a portion of the vision data that captures the corresponding additional vehicle.

5. The method of claim 1, wherein training the machine learning model comprises:

processing, using the machine learning model, the training instance inputs of the training instances to generate predicted outputs;

generating losses based on the predicted outputs and the supervised training instance outputs of the training instances; and

updating the machine learning model based on the generated losses.

6. The method of claim 1, wherein the instance of additional vehicle data comprises one or more yaw parameters and the one or more sensors of the corresponding additional vehicle comprise at least a yaw rate sensor.

7. The method of claim 6, wherein the additional vehicle data is determined based on monitoring of a controller area network of the corresponding additional vehicle.

8. The method of claim 1, wherein the instance of additional vehicle data comprises one or both of velocity and acceleration of the corresponding additional vehicle.

9. The method of claim 1, wherein the instance of additional vehicle data comprises yaw parameters and one or both of velocity and acceleration.

10. The method of claim 1, further comprising:

processing, by at least one processor of a given autonomous vehicle of the at least one autonomous vehicle, given vision data using the trained machine learning model, the given vision data captured by a given vision component of the given autonomous vehicle;

generating, based on the processing, a predicted state of a given additional vehicle captured by the given vision data; and

controlling the given autonomous vehicle based on the predicted state.

11. The method of claim 10, wherein the given vision data is a subgroup of vision data captured during a sensing cycle of the vision component, and further comprising:

generating the subgroup of vision data based on determining the subgroup corresponds to the given additional vehicle.

12. The method of claim 11, wherein determining the subgroup corresponds to the given additional vehicle comprises:

processing the vision data using an additional object detection and classification model; and

determining the subgroup corresponds to the given additional vehicle based on output generated based on processing the vision data using the additional object detection and classification model.

13. A method of controlling an autonomous vehicle using a trained machine learning model, the method implemented by one or more processors of the autonomous vehicle, and the method comprising:

processing given vision data using the trained machine learning model, the given vision data captured by a vision component of the autonomous vehicle;

generating, based on the processing, a predicted state of at least one dynamic property of a given additional vehicle captured by the given vision data; and

controlling the autonomous vehicle based on the predicted state;

wherein the trained machine learning model is trained based on a plurality of training instances that each includes;

training instance input that is based on a corresponding instance of vision data from a vision component of a corresponding autonomous vehicle; and

supervised training instance output that is based on a corresponding instance of additional vehicle data, wherein the corresponding instance of additional vehicle data is based on one or more sensors of a corresponding additional vehicle that is captured by the corresponding instance of vision data.

14. The method of claim 13, wherein the given vision data is a subgroup of vision data captured during a sensing cycle of the vision component, and further comprising:

generating the subgroup of vision data based on determining the subgroup corresponds to the given additional vehicle.

15. The method of claim 13, wherein determining the subgroup corresponds to the given additional vehicle comprises:

processing the vision data using an additional object detection and classification model; and

determining the subgroup corresponds to the given additional vehicle based on output generated based on processing the vision data using the additional object detection and classification model.

16. The method of claim 13, wherein controlling the autonomous vehicle comprises controlling speed and/or direction of the autonomous vehicle.

17. The method of claim 13, wherein the at least one property comprises one or both of a yaw rate and a yaw direction of the additional vehicle.

18. The method of claim 13, wherein the at least one property comprises one or both of velocity and acceleration of the additional vehicle.

19. The method of claim 13, wherein the at least one property comprises a yaw parameter and one or both of velocity and acceleration of the additional vehicle.