System and method for real world autonomous vehicle trajectory simulation

• US 10,739,775 B2
• Filed: 10/28/2017
• Issued: 08/11/2020
• Est. Priority Date: 10/28/2017
• Status: Active Grant

• Current Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Original Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Inventors: Sun, Xing, Lin, Wutu, Liu, Liu, Ma, Kai-Chieh, Xuan, Zijie, Zhao, Yufei
• Primary Examiner(s): Lonsberry, Hunter B
• Assistant Examiner(s): Martinez Borrero, Luis A
• Application Number:
  US15/796,765
• Publication Number:
  US 20190129436A1
• Time in Patent Office: 1,018 Days
• CPC Class Codes: G05B 13/048 using a predictor
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0221 involving a learning process
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

A system and method for real world autonomous vehicle trajectory simulation are disclosed. A particular embodiment includes: receiving training data from a real world data collection system; obtaining ground truth data corresponding to the training data; performing a training phase to train a plurality of trajectory prediction models; and performing a simulation or operational phase to generate a vicinal scenario for each simulated vehicle in an iteration of a simulation, the vicinal scenarios corresponding to different locations, traffic patterns, or environmental conditions being simulated, provide vehicle intention data corresponding to a data representation of various types of simulated vehicle or driver intentions, generate a trajectory corresponding to perception data and the vehicle intention data, execute at least one of the plurality of trained trajectory prediction models to generate a distribution of predicted vehicle trajectories for each of a plurality of simulated vehicles of the simulation based on the vicinal scenario and the vehicle intention data, select at least one vehicle trajectory from the distribution based on pre-defined criteria, and update a state and trajectory of each of the plurality of simulated vehicles based on the selected vehicle trajectory from the distribution.

Claims

1. A system comprising:

a data processor in a computing system;

a data collection system, executable by the data processor of the computing system, to collect training data comprising perception or sensor data and ground truth data, and perform a training phase to train a plurality of trajectory prediction models with the training data to produce a plurality of trained trajectory prediction models, the data collection system comprising sensor devices, the sensor devices comprising at least one image generating device, the sensor devices, installed at various traffic locations or installed on moving test vehicles, collecting perception or sensor data from the various traffic locations or from the moving test vehicles, the plurality of trained trajectory prediction models comprising at least one trained trajectory prediction model configured to model a variable level of simulated driver aggressiveness, the collected training data being transferred to the data collection system;

a vicinal scene data generator to generate, by the data processor, a different vicinal scenario for each simulated vehicle in an iteration of a simulation;

a memory for storage of vehicle intention data representing simulated vehicle or driver intentions for a plurality of different vehicle actions and behaviors; and

a trajectory simulation module, executable by the data processor of the computing system, the trajectory simulation module being configured to perform a simulation or operational phase to;

generate, by the data processor, a trajectory corresponding to the collected perception data and the vehicle intention data, execute, by the data processor, at least one of the plurality of trained trajectory prediction models to generate a predicted simulated vehicle trajectory for each of a plurality of simulated vehicles of the simulation based on a corresponding generated vicinal scenario and the vehicle intention data, the predicted simulated vehicle trajectory comprising data indicative of a degree of likelihood or probability that a particular simulated vehicle will traverse the corresponding predicted simulated vehicle trajectory, use the predicted simulated vehicle trajectory, and update, by the data processor, a state and trajectory of each of the plurality of simulated vehicles based on the predicted simulated vehicle trajectory, the plurality of trained trajectory prediction models being used for configuring a control system in an autonomous vehicle.

2. The system of claim 1 wherein the trajectory simulation operation comprises machine learnable components.

3. The system of claim 1 wherein the trajectory simulation operation is performed over multiple iterations.

4. The system of claim 1 wherein the trajectory simulation operation being further configured to receive an array of sensor information gathered at the various traffic locations by the data collection system.

5. The system of claim 1 wherein the different vicinal scenarios correspond to different locations, traffic patterns, or environmental conditions being simulated.

6. The system of claim 1 wherein the trajectory simulation module being further configured to execute, by the data processor, at least one of the plurality of trained trajectory prediction models to generate a distribution of predicted simulated vehicle trajectories for each of a plurality of simulated vehicles of the simulation based on a corresponding generated vicinal scenario and the vehicle intention data, the distribution of predicted simulated vehicle trajectories comprising data indicative of a degree of likelihood or probability that a particular simulated vehicle will traverse a corresponding trajectory of the distribution of predicted simulated vehicle trajectories, select at least one simulated vehicle trajectory from the distribution of predicted simulated vehicle trajectories based on pre-defined criteria, and update, by the data processor, a state and trajectory of each of the plurality of simulated vehicles based on the selected simulated vehicle trajectory from the distribution of predicted simulated vehicle trajectories.

7. The system of claim 1 wherein the variable level of simulated driver aggressiveness is modeled from behaviors of human drivers collected by the data collection system.

8. A method comprising:

receiving training data, by a data processor in a computing system, from a data collection system, the data collection system comprising sensor devices, the sensor devices comprising at least one image generating device, the sensor devices, installed at various traffic locations or installed on moving test vehicles, collecting perception or sensor data from the various traffic locations or from the moving test vehicles;

obtaining ground truth data corresponding to the training data;

transferring the collected training data to the data collection system;

using the data collection system, executed by the data processor of the computing system, and the training data to train a plurality of trajectory prediction models to produce a plurality of trained trajectory prediction models, the plurality of trained trajectory prediction models comprising at least one trained trajectory prediction model configured to model a variable level of simulated driver aggressiveness; and

performing, by the data processor, a simulation or operational phase to generate a different vicinal scenario for each simulated vehicle in an iteration of a simulation, provide vehicle intention data corresponding to a data representation of various types of simulated vehicle or driver intentions for a plurality of different vehicle actions and behaviors, generate, by the data processor, a trajectory corresponding to the collected perception data and the vehicle intention data, execute, by the data processor, at least one of the plurality of trained trajectory prediction models to generate a predicted simulated vehicle trajectory for each of a plurality of simulated vehicles of the simulation based on a corresponding generated vicinal scenario and the vehicle intention data, the predicted simulated vehicle trajectory comprising data indicative of a degree of likelihood or probability that a particular simulated vehicle will traverse the corresponding predicted simulated vehicle trajectory, use the predicted simulated vehicle trajectory, and update, by the data processor, a state and trajectory of each of the plurality of simulated vehicles based on the predicted simulated vehicle trajectory, the plurality of trained trajectory prediction models being used for configuring a control system in an autonomous vehicle.

9. The method of claim 8 wherein the method is machine learnable.

10. The method of claim 8 wherein the method is performed over multiple iterations.

11. The method of claim 8 comprising receiving an array of sensor information gathered at the various traffic locations by the data collection system.

12. The method of claim 8 wherein the different vicinal scenarios correspond to different locations, traffic patterns, or environmental conditions being simulated.

13. The method of claim 8 wherein the simulation or operational phase further to execute, by the data processor, at least one of the plurality of trained trajectory prediction models to generate a distribution of predicted simulated vehicle trajectories for each of a plurality of simulated vehicles of the simulation based on a corresponding generated vicinal scenario and the vehicle intention data, the distribution of predicted simulated vehicle trajectories comprising data indicative of a degree of likelihood or probability that a particular simulated vehicle will traverse a corresponding trajectory of the distribution of predicted simulated vehicle trajectories, select at least one simulated vehicle trajectory from the distribution of predicted simulated vehicle trajectories based on pre-defined criteria, and update, by the data processor, a state and trajectory of each of the plurality of simulated vehicles based on the selected simulated vehicle trajectory from the distribution of predicted simulated vehicle trajectories.

14. The method of claim 8 wherein the variable level of simulated driver aggressiveness is modeled from behaviors of human drivers collected by the data collection system.

15. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:

receive training data, by a data processor in a computing system, from a data collection system, the data collection system comprising sensor devices, the sensor devices comprising at least one image generating device, the sensor devices, installed at various traffic locations or installed on moving test vehicles, collecting perception or sensor data from the various traffic locations or from the moving test vehicles;

obtain ground truth data corresponding to the training data;

transfer the collected training data to the data collection system;

use the data collection system, executed by the data processor of the computing system, and the training data to train a plurality of trajectory prediction models to produce a plurality of trained trajectory prediction models, the plurality of trained trajectory prediction models comprising at least one trained trajectory prediction model configured to model a variable level of simulated driver aggressiveness; and

perform, by the data processor, a simulation or operational phase to generate a different vicinal scenario for each simulated vehicle in an iteration of a simulation, provide vehicle intention data corresponding to a data representation of various types of simulated vehicle or driver intentions for a plurality of different vehicle actions and behaviors, generate, by the data processor, a trajectory corresponding to the collected perception data and the vehicle intention data, execute, by the data processor, at least one of the plurality of trained trajectory prediction models to generate a predicted simulated vehicle trajectory for each of a plurality of simulated vehicles of the simulation based on a corresponding generated vicinal scenario and the vehicle intention data, the predicted simulated vehicle trajectory comprising data indicative of a degree of likelihood or probability that a particular simulated vehicle will traverse the corresponding predicted simulated vehicle trajectory, use the predicted simulated vehicle trajectory, and update, by the data processor, a state and trajectory of each of the plurality of simulated vehicles based on the predicted simulated vehicle trajectory, the plurality of trained trajectory prediction models being used for configuring a control system in an autonomous vehicle.

16. The non-transitory machine-useable storage medium of claim 15 wherein the instructions comprise machine learnable components.

17. The non-transitory machine-useable storage medium of claim 15 wherein the instructions are executed over multiple iterations.

18. The non-transitory machine-useable storage medium of claim 15 wherein the instructions being further configured to receive an array of sensor information gathered at the various traffic locations by the data collection system.

19. The non-transitory machine-useable storage medium of claim 15 wherein the different vicinal scenarios correspond to different locations, traffic patterns, or environmental conditions being simulated.

20. The non-transitory machine-useable storage medium of claim 15 being further configured to execute, by the data processor, at least one of the plurality of trained trajectory prediction models to generate a distribution of predicted simulated vehicle trajectories for each of a plurality of simulated vehicles of the simulation based on a corresponding generated vicinal scenario and the vehicle intention data, the distribution of predicted simulated vehicle trajectories comprising data indicative of a degree of likelihood or probability that a particular simulated vehicle will traverse a corresponding trajectory of the distribution of predicted simulated vehicle trajectories, select at least one simulated vehicle trajectory from the distribution of predicted simulated vehicle trajectories based on pre-defined criteria, and update, by the data processor, a state and trajectory of each of the plurality of simulated vehicles based on the selected simulated vehicle trajectory from the distribution of predicted simulated vehicle trajectories.