Method and apparatus for scenario generation and parametric sweeps for the development and evaluation of autonomous driving systems

• US 10,345,811 B2
• Filed: 11/14/2017
• Issued: 07/09/2019
• Est. Priority Date: 11/14/2017
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Phillips, Matthew E., Bergstedt, Dylan T., Thor, Nandan, Choe, Jaehoon, Daily, Michael J.
• Primary Examiner(s): Paige, Tyler D
• Application Number:
  US15/811,878
• Publication Number:
  US 20190146492A1
• Time in Patent Office: 602 Days
• Field of Search: 701 23
• CPC Class Codes: G05B 13/041 in which a variable is automatically adjusted to optimise the performance
• CPC Class Codes: G05B 17/02 electric
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G07C 5/085 using electronic data carriers
• CPC Class Codes: G08G 1/16 Anti-collision systems road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08
• CPC Class Codes: G09B 19/16 Control of vehicles or other craft simulators G09B9/02
• CPC Class Codes: G09B 19/167 Control of land vehicles
• CPC Class Codes: G09B 9/04 for teaching control of land vehicles

Abstract

The present application generally relates to methods and apparatus for evaluating driving performance under a plurality of driving scenarios and conditions. More specifically, the application teaches a method and apparatus for testing a driving scenario repetitively while altering a parametric variation, such as fog level, in order to evaluate driving system performance under changing conditions.

Claims

1. An apparatus comprising:

a sensor interface for generating sensor data for coupling to a vehicle control system;

a control system interface for receiving control data from the vehicle control system;

a memory for storing a first scenario wherein the first scenario is associated with a first parametric variation and a second parametric variation; and

a simulator for simulating a first driving environment in response to the first scenario, the first parametric variation and the control data and assigning a first performance metric in response to the simulation of the first driving environment, the simulator further operative to simulate a second driving environment in response to the first scenario, the second parametric variation and the control data and assigning a second performance metric in response to the simulation of the second driving environment.

2. The apparatus of claim 1 wherein the first parametric variation and the second parametric variation relate to intensity of weather.

3. The apparatus of claim 1 wherein the memory is operative to store a second scenario associated with the first parametric variation and the second parametric variation.

4. The apparatus of claim 1 wherein the apparatus is implemented in software.

5. The apparatus of claim 1 wherein the apparatus is implemented in hardware.

6. The apparatus of claim 1 wherein the simulator is further operative to generate a success rate in response to the first performance metric and the second performance metric.

7. The apparatus of claim 1 wherein the vehicle control system is an autonomous vehicle control system.

8. A method comprising:

receiving a driving scenario, a first parametric variation and a second parametric variation;

simulating the driving scenario using the first parametric variation;

evaluating a first driver performance in response to the driving scenario, the first parametric variation and a first control data;

assigning a first performance metric in response to the first driver performance;

simulating the driving scenario using the second parametric variation;

evaluating a second driver performance in response to the driving scenario, the second parametric variation and a second control data; and

assigning a second performance metric in response to the second driver performance.

9. The method of claim 8 further comprising assigning an overall performance metric in response to the first performance metric and the second performance metric.

10. The method of claim 8 wherein the first parametric variation and the second parametric variation relate to intensity of weather.

11. The method of claim 8 wherein the first parametric variation and the second parametric variation relate to a velocity of a vehicle within the driving scenario.

12. The method of claim 8 wherein the first parametric variation and the second parametric variation relate to a number of vehicles within the driving scenario.

13. The method of claim 8 wherein the first parametric variation and the second parametric variation relate to velocity of a pedestrian within the driving scenario.

14. The method of claim 8 wherein the first parametric variation and the second parametric variation relate to a visibility within the driving scenario.

15. The method of claim 8 wherein further comprising receiving a third parametric variation, simulating the driving scenario using the third parametric variation and assigning a third performance metric in response to a third driver performance.

16. The method of claim 15 further comprising assigning an overall performance metric in response to the first performance metric, the second performance metric and the third performance metric.

17. The method of claim 8 wherein the first parametric variation and the second parametric variation relate to sensor performance within the driving scenario.