Evaluation framework for decision making of autonomous driving vehicle

• US 10,421,460 B2
• Filed: 11/09/2016
• Issued: 09/24/2019
• Est. Priority Date: 11/09/2016
• Status: Active Grant

• Current Assignee: Baidu USA LLC (Baidu Incorporated)
• Original Assignee: Baidu USA LLC (Baidu Incorporated)
• Inventors: Jiang, Yifei, Li, Dong, Tao, Jiaming, Hu, Jiangtao, Li, Liyun, Yang, Guang, Wang, Jingao
• Primary Examiner(s): Shudy, Angelina
• Application Number:
  US15/347,659
• Publication Number:
  US 20180127000A1
• Time in Patent Office: 1,049 Days
• Field of Search: 701 23
• CPC Class Codes: B60K 2360/175 Autonomous driving
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60W 2050/0075 Automatic parameter input, automatic initialising or calibrating means
• CPC Class Codes: B60W 2050/0088 Adaptive recalibration
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2554/4029 Pedestrians
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 30/18154 Approaching an intersection
• CPC Class Codes: B60W 40/09 Driving style or behaviour
• CPC Class Codes: B60W 50/045 Monitoring control system parameters
• CPC Class Codes: B60W 60/0016 of the vehicle or its occupants
• CPC Class Codes: G05D 1/0061 for transition from automatic pilot to manual pilot and vice versa
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

In one embodiment, systems and methods are disclosed for evaluating autonomous driving vehicle (ADV) driving decisions. A driving scenario is selected, such as a route or destination or type of driving condition. ADV planning and control modules are turned off and do not control the ADV. As a user drives the ADV, sensors detect and periodically log a plurality of objects external to the ADV. Driving control inputs of the human driver are also logged periodically. An ADV driving decision module generates driving decisions with respect to each object detected by the sensors. The ADV driving decisions are logged, but are not used to control the ADV. An ADV driving decision is identified in the logs, and a corresponding human driving decision is extracted, graded, and compared to the ADV driving decision. The ADV driving decision can be graded using the logs and graded human driving decision.

Claims

1. A computer-implemented method, comprising:

receiving a plurality of autonomous driving vehicle (ADV) logs;

extracting a human driving decision corresponding to an ADV driving decision using the plurality of ADV logs;

grading the human driving decision in accordance with a plurality of grading metrics; and

generating a grade for the ADV driving decision that is based at least in part on the grade of the human driving decision and a comparison of the human driving decision to the ADV driving decision, wherein the grade for the ADV driving decision is utilized to modify algorithms for subsequent ADV driving.

2. The method of claim 1, wherein the plurality ADV logs comprise a log of a plurality of ADV driving decisions made by a decision module of the ADV, a perception log comprising a log of a plurality of object sensor outputs, and a driving log comprising a log of a plurality of driving control inputs, wherein the plurality of driving control inputs were captured from the ADV while the ADV was driven by a user.

3. The method of claim 2, wherein a driving control input of the plurality of driving control inputs comprises one of an input amount to an acceleration control, an input amount to a brake control, or an input amount to a steering control.

4. The method of claim 2, wherein extracting the human driving decision corresponding to the ADV driving decision, with respect to an object, comprises one or more of:

determining a time at which the ADV decision was made;

analyzing the perception log for a time before and after the ADV driving decision;

or

analyzing the driving log for a period of time before, and a period of time after, occurrence of the ADV driving decision being extracted;

wherein analyzing comprises determining a relationship between one or more of the plurality of driving control inputs and a location or distance of the object with respect to the ADV.

5. The method of claim 2, wherein an object sensor comprises one or more of light detection and ranging (LIDAR), radio detection and ranging (RADAR), global positioning system (GPS), or a camera, and wherein an object sensor output of the plurality of sensor object outputs comprises one or more of a distance to an object from the ADV, a speed of an object external to the ADV, a speed of the ADV, an orientation of an object external to the ADV with respect to the ADV, a direction of an object external to the ADV, a direction of the ADV, or a location of the ADV.

6. The method of claim 1, wherein the ADV driving decision is specific to an object external to the ADV and the ADV driving decision comprises one of:

overtake the object, yield to the object, stop the ADV at least a minimum distance from the object, or ignore the object.

7. The method of claim 1, wherein a grading metric of the plurality of grading metrics comprises a safety metric, a comfort metric, a rule compliance metric, or a time efficiency metric.

8. The method of claim 1, wherein grading the human driving decision comprises:

analyzing one or more driving inputs or one or more object sensor outputs that were logged after the human driving decision was logged, to determine a metric for safety, comfort, rule compliance, or time efficiency of the human driving decision.

9. The method of claim 1, wherein grading the ADV driving decision comprises:

estimating one or more driving inputs or one or more object sensor outputs, after the ADV driving decision, to determine a metric for safety, comfort, rule compliance, or time efficiency of the ADV driving decision.

10. The medium of claim 9, wherein grading the ADV driving decision comprises:

estimating one or more driving inputs or one or more object sensor outputs, after the ADV driving decision, to determine a metric for safety, comfort, rule compliance, or time efficiency of the ADV driving decision.

11. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising:

receiving a plurality of autonomous driving vehicle (ADV) logs;

extracting a human driving decision corresponding to an ADV driving decision using the plurality of ADV logs;

grading the human driving decision in accordance with a plurality of grading metrics; and

generating a grade for the ADV driving decision that is based at least in part on the grade of the human driving decision and a comparison of the human driving decision to the ADV driving decision, wherein the grade for the ADV driving decision is utilized to modify algorithms for subsequent ADV driving.

12. The medium of claim 11, wherein the plurality ADV logs comprise a log of a plurality of ADV driving decisions made by a decision module of the ADV, a perception log comprising a log of a plurality of object sensor outputs, and a driving log comprising a log of a plurality of driving control inputs, wherein the plurality of driving control inputs were captured from the ADV while the ADV was driven by a user.

13. The medium of claim 12, wherein a driving control input of the plurality of driving control inputs comprises one of an input amount to an acceleration control, an input amount to a brake control, or an input amount to a steering control.

14. The medium of claim 12, wherein extracting the human driving decision corresponding to the ADV driving decision, with respect to an object, comprises one or more of:

determining a time at which the ADV decision was made;

analyzing the perception log for a time before and after the ADV driving decision;

or

analyzing the driving log for a period of time before, and after of time after, occurrence of the ADV driving decision being extracted;

wherein analyzing comprises determining a relationship between one or more of the plurality of driving control inputs and a location or distance of the object with respect to the ADV.

15. The medium of claim 12, wherein an object sensor comprises one or more of light detection and ranging (LIDAR), radio detection and ranging (RADAR), global positioning system (GPS), or a camera, and wherein an object sensor output of the plurality of object sensor outputs comprises one or more of a distance to an object from the ADV, a speed of an object external to the ADV, a speed of the ADV, an orientation of an object external to the ADV with respect to the ADV, a direction of an object external to the ADV, a direction of the ADV, or a location of the ADV.

16. The medium of claim 11, wherein the ADV driving decision is specific to an object external to the ADV and the ADV driving decision comprises one of:

overtake the object, yield to the object, stop the ADV at least a minimum distance from the object, or ignore the object.

17. The medium of claim 11, wherein a grading metric of the plurality of grading metrics comprises a safety metric, a comfort metric, a rule compliance metric, or a time efficiency metric.

18. The medium of claim 11, wherein grading the human driving decision comprises:

analyzing one or more driving inputs or one or more object sensor outputs that were logged after the human driving decision was logged, to determine a metric for safety, comfort, rule compliance, or time efficiency of the human driving decision.

19. A data processing system, comprising:

a processor; and

a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including;

receiving a plurality of autonomous driving vehicle (ADV) logs,

extracting a human driving decision corresponding to an ADV driving decision using the plurality of ADV logs,

grading the human driving decision in accordance with a plurality of grading metrics, and

generating a grade for the ADV driving decision that is based at least in part on the grade of the human driving decision and a comparison of the human driving decision to the ADV driving decision, wherein the grade for the ADV driving decision is utilized to modify algorithms for subsequent ADV driving.

20. The data processing system of claim 19, wherein the plurality ADV logs comprise a log of a plurality of ADV driving decisions made by a decision module of the ADV, a perception log comprising a log of a plurality of object sensor outputs, and a driving log comprising a log of a plurality of driving control inputs, wherein the plurality of driving control inputs were captured from the ADV while the ADV was driven by a user.

21. The data processing system of claim 20, wherein a driving control input of the plurality of driving control inputs comprises one of an input amount to an acceleration control, an input amount to a brake control, or an input amount to a steering control.

22. The data processing system of claim 20, wherein extracting the human driving decision corresponding to the ADV driving decision, with respect to an object, comprises one or more of:

determining a time at which the ADV decision was made;

analyzing the perception log for a time before and after the ADV driving decision;

or

analyzing the driving log for a period of time before, and a period of time after, occurrence of the ADV driving decision being extracted;

wherein analyzing comprises determining a relationship between one or more of the plurality of driving control inputs and a location or distance of the object with respect to the ADV.

23. The data processing system of claim 20, wherein an object sensor comprises one or more of light detection and ranging (LIDAR), radio detection and ranging (RADAR), global positioning system (GPS), or a camera, and wherein an object sensor output of the plurality of object sensor outputs comprises one or more of a distance to an object from the ADV, a speed of an object external to the ADV, a speed of the ADV, an orientation of an object external to the ADV with respect to the ADV, a direction of an object external to the ADV, a direction of the ADV, or a location of the ADV.

24. The data processing system of claim 19, wherein the ADV driving decision is specific to an object external to the ADV and the ADV driving decision comprises one of:

overtake the object, yield to the object, stop the ADV at least a minimum distance from the object, or ignore the object.

25. The data processing system of claim 19, wherein a grading metric of the plurality of grading metrics comprises a safety metric, a comfort metric, a rule compliance metric, or a time efficiency metric.

26. The data processing system of claim 19, wherein grading the human driving decision comprises:

analyzing one or more driving inputs or one or more object sensor outputs that were logged after the human driving decision was logged, to determine a metric for safety, comfort, rule compliance, or time efficiency of the human driving decision.

27. The data processing system of claim 19, wherein grading the ADV driving decision comprises:

estimating one or more driving inputs or one or more object sensor outputs, after the ADV driving decision, to determine a metric for safety, comfort, rule compliance, or time efficiency of the ADV driving decision.