Software validation for autonomous vehicles

• US 10,831,636 B2
• Filed: 01/08/2018
• Issued: 11/10/2020
• Est. Priority Date: 01/08/2018
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Morley, Franklin, Baror, Omer, Fairfield, Nathaniel, Crary, Miner
• Primary Examiner(s): Do, Chat C
• Assistant Examiner(s): Slachta, Douglas M
• Application Number:
  US15/864,239
• Publication Number:
  US 20190213103A1
• Time in Patent Office: 1,037 Days
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06F 11/3608 using formal methods, e.g. model checking, abstract interpretation theorem proving G06N5/013
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

Aspects of the disclosure relate to validating autonomous control software for operating a vehicle autonomously. For instance, the autonomous control software is run through a driving scenario to observe an outcome for the autonomous control software. A validation model is run through the driving scenario a plurality of times to observe an outcome for the model for each of the plurality of times. Whether the software passed the driving scenario is determined based on whether the outcome for the software indicates that a virtual vehicle under control of the software collided with another object during the single time. Whether the validation model passed the driving scenario is determined based on whether the outcome for the model indicates that a virtual vehicle under control of the model collided with another object in any one of the plurality of times. The software is validated based on the determinations.

Claims

1. A method of validating autonomous control software for operating a vehicle in an autonomous driving mode, the method comprising:

running the autonomous control software through a driving scenario a single time to observe an outcome of the scenario for the autonomous control software;

running a validation model that is different from the autonomous control software through the driving scenario once for each of a plurality of potential responses to the driving scenario to observe an outcome of the scenario for the validation model for each of the plurality of potential responses;

determining whether the autonomous control software passed the driving scenario based on whether the outcome of the scenario for the autonomous control software indicates that a virtual vehicle under control of the autonomous control software collided with another object during the single time;

determining whether the validation model passed the driving scenario based on whether the outcome of the scenario for the validation model indicates that a virtual vehicle under control of the validation model collided with another object in any one of the plurality of potential responses; and

validating the autonomous control software based on a comparison of the determinations.

2. The method of claim 1, wherein the driving scenario is one of a plurality of driving scenarios for a given category of driving scenarios, and wherein validating the autonomous control software is based on evaluating performance of the autonomous control software and the validation model over all of the plurality of driving scenarios for the given category.

3. The method of claim 2, further comprising comparing a number of the driving scenarios passed by the autonomous control software to a number of scenarios passed at least once in a plurality of potential responses, and wherein validating the autonomous control software is further based on the comparison.

4. The method of claim 1, wherein the plurality of potential responses includes two or more of:

accelerating;

braking;

swerving right;

swerving left;

braking and swerving right;

braking and swerving left;

accelerating and swerving right;

accelerating and swerving left;

waiting a fixed amount of time and then accelerating;

waiting a fixed amount of time and then braking;

waiting a fixed amount of time and then swerving right;

waiting a fixed amount of time and then swerving left;

waiting a fixed amount of time and then braking and swerving right;

waiting a fixed amount of time and then braking and swerving left;

waiting a fixed amount of time and then accelerating and swerving right;

waiting a fixed amount of time and then accelerating and swerving left; and

making no change to speed, acceleration, braking, and direction.

5. The method of claim 1, wherein each of the plurality of potential responses to the driving scenario are different, such that a first one of the plurality includes a braking response, a second one of the plurality includes a swerving right response, a third one of the plurality includes a swerving left response, a fourth one of the plurality includes a braking and swerving right response, and a fifth one of the plurality includes a braking and swerving left response.

6. The method of claim 1, wherein determining whether the autonomous control software passed the driving scenario is further based on a severity level of any collision with the another object.

7. The method of claim 6, wherein validating the autonomous control software is further based on the severity level.

8. The method of claim 1, wherein determining whether the validation model passed the driving scenario is further based on a severity level of any collision with the another object.

9. The method of claim 8, wherein validating the autonomous control software is further based on the severity level.

10. The method of claim 1, further comprising, identifying a handover time for giving the autonomous control software or the validation model control of the virtual vehicle in the scenario corresponding to a predetermined number of seconds within the scenario before the virtual vehicle would collide with the another object if both objects continued on respective current trajectories in the scenario, and wherein the handover time is used to run the validation model through the driving scenario and to run the autonomous control software through the driving scenario.

11. The method of claim 10, wherein the handover time is used to run the autonomous control software through the driving scenario.

12. The method of claim 10, further comprising, providing the scenario and the handover time to a human operator for confirmation of the handover time.

13. A system for validating autonomous control software for operating a vehicle in an autonomous driving mode, the system comprising one or more computing devices having one or more processors configured to:

run the autonomous control software through a driving scenario a single time to observe an outcome of the scenario for the autonomous control software;

run a validation model that is different from the autonomous control software through the driving scenario once for each of a plurality of potential responses to the driving scenario to observe an outcome of the scenario for the validation model for each of the plurality of potential responses;

determine whether the autonomous control software passed the driving scenario based on whether the outcome of the scenario for the autonomous control software indicates that a virtual vehicle under control of the autonomous control software collided with another object during the single time;

determine whether the validation model passed the driving scenario based on whether the outcome of the scenario for the validation model indicates that a virtual vehicle under control of the validation model collided with another object in any one of the plurality of potential responses; and

validate the autonomous control software based on a comparison of the determinations.

14. The system of claim 13, wherein the driving scenario is one of a plurality of driving scenarios for a given category of driving scenarios, and wherein validating the autonomous control software is based on evaluating performance of the autonomous control software and the validation model over all of the plurality of driving scenarios for the given category.

15. The system of claim 14, wherein the one or more processors are further configured to compare a number of the driving scenarios passed by the autonomous control software to a number of scenarios passed at least once in a plurality of potential responses, and wherein validating the autonomous control software is further based on the comparison.

16. The system of claim 13, wherein the one or more processors are further configured to determine whether the autonomous control software passed the driving scenario further based on a severity level of any collision with the another object.

17. The system of claim 13, wherein the one or more processors are further configured to determine whether the validation model passed the driving scenario further based on a severity level of any collision with the another object.

18. The system of claim 13, wherein the one or more processors are further configured to identify a handover time for giving the autonomous control software or the validation model control of the virtual vehicle in the scenario corresponding to a predetermined number of seconds within the scenario before the virtual vehicle would collide with the another object if both objects continued on respective current trajectories in the scenario, and wherein the handover time is used to run the validation model through the driving scenario and to run the autonomous control software through the driving scenario.

19. The system of claim 13, further comprising the autonomous control software.

20. The system of claim 13, further comprising the validation model.