Systems and methods to test an autonomous vehicle

• US 10,431,023 B1
• Filed: 09/06/2017
• Issued: 10/01/2019
• Est. Priority Date: 08/21/2017
• Status: Active Grant

• Current Assignee: Uatc, LLC (Uber Technologies, Inc.)
• Original Assignee: Uber Technologies, Inc.
• Inventors: Watson, Adam, MacIntosh, Chad David, Bembic, Anthony, Leja, Joseph Richard
• Primary Examiner(s): Beaulieu, Yonel
• Application Number:
  US15/697,044
• Time in Patent Office: 755 Days
• CPC Class Codes: G01M 17/007 Wheeled or endless-tracked vehicles G01M17/08 takes precedence
• CPC Class Codes: G01M 17/0078 Shock-testing of vehicles
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G07C 5/08 Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
• CPC Class Codes: G07C 5/0841 Registering performance data recording measured values G01D; information storage G11B

Abstract

The present disclosure provides systems and methods to test an autonomous vehicle. In particular, the systems and methods of the present disclosure can receive, from one or more test nodes of a preconfigured test track, log data indicating positions of elements of the test track over a period of time. Log data indicating parameters of an autonomous vehicle over the period of time can be received from the autonomous vehicle. The log data indicating the positions of the elements of the test track over the period of time can be compared with the log data indicating the parameters of the autonomous vehicle over the period of time to determine a performance metric of the autonomous vehicle on the test track over the period of time.

Claims

1. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising:

comparing log data indicating positions of elements of a test track over a first period of time with log data indicating positions of the elements of the test track over a second period of time;

comparing log data indicating parameters of an autonomous vehicle over the first period of time with log data indicating the parameters of the autonomous vehicle over the second period of time; and

determining an effect of operations of the autonomous vehicle over a period of time between the first period of time and the second period of time, the effect caused by modifying software configured to control operations of the autonomous vehicle, wherein determining the effect comprises determining a difference in a measure of smoothness in transition from a first state to a second state of the autonomous vehicle.

2. The one or more non-transitory computer-readable media of claim 1, wherein the elements of the test track comprise at least one of a simulated vehicle, a simulated pedestrian, or a simulated cyclist.

3. The one or more non-transitory computer-readable media of claim 1, wherein the parameters of the autonomous vehicle comprise one or more of a rate of acceleration of the autonomous vehicle, a rate of deceleration of the autonomous vehicle, a speed of the autonomous vehicle, or a heading of the autonomous vehicle.

4. The one or more non-transitory computer-readable media of claim 1, wherein determining the effect comprises determining the effect based at least in part on feedback, received from a user device, describing performance of the autonomous vehicle over the first period of time and the second period of time.

5. The one or more non-transitory computer-readable media of claim 1, wherein:

the log data indicating the positions of the elements of the test track over the first period of time indicates a position or status of one or more actuators associated with the elements of the test track over the first period of time; and

the log data indicating the positions of the elements of the test track over the second period of time indicates a position or status of the one or more actuators associated with the elements of the test track over the second period of time.

6. The one or more non-transitory computer-readable media of claim 1, wherein:

the log data indicating the positions of the elements of the test track over the first period of time comprises data from one or more global positioning system (GPS) receivers attached to the elements; and

the log data indicating the positions of the elements of the test track over the second period of time comprises data from the one or more GPS receivers attached to the elements.

7. The one or more non-transitory computer-readable media of claim 1, wherein:

the first state comprises a state in which the autonomous vehicle is moving; and

the second state comprises a state in which the autonomous vehicle is stationary.

8. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising:

comparing log data indicating positions of elements of a test track over a first period of time with log data indicating positions of the elements of the test track over a second period of time;

comparing log data indicating parameters of an autonomous vehicle over the first period of time with log data indicating the parameters of the autonomous vehicle over the second period of time; and

determining an effect of operations of the autonomous vehicle over a period of time between the first period of time and the second period of time, the effect caused by modifying software configured to control the operations of the autonomous vehicle, wherein determining the effect of the operations comprises determining the effect of the operations based at least in part on feedback, received from a user device, describing performance of the autonomous vehicle over the first period of time and the second period of time.

9. The one or more non-transitory computer-readable media of claim 8, wherein the elements of the test track comprise at least one of a simulated vehicle, a simulated pedestrian, or a simulated cyclist.

10. The one or more non-transitory computer-readable media of claim 8, wherein the parameters of the autonomous vehicle comprise one or more of a rate of acceleration of the autonomous vehicle, a rate of deceleration of the autonomous vehicle, a speed of the autonomous vehicle, or a heading of the autonomous vehicle.

11. The one or more non-transitory computer-readable media of claim 8, wherein:

the log data indicating the positions of the elements of the test track over the first period of time indicates a position or status of one or more actuators associated with the elements of the test track over the first period of time; and

the log data indicating the positions of the elements of the test track over the second period of time indicates a position or status of the one or more actuators associated with the elements of the test track over the second period of time.

12. The one or more non-transitory computer-readable media of claim 8, wherein:

the log data indicating the positions of the elements of the test track over the first period of time comprises data from one or more global positioning system (GPS) receivers attached to the elements; and

the log data indicating the positions of the elements of the test track over the second period of time comprises data from the one or more GPS receivers attached to the elements.

13. The one or more non-transitory computer-readable media of claim 8, wherein:

the log data indicating the parameters of the autonomous vehicle over the first period of time comprises data from a global positioning system (GPS) receiver of the autonomous vehicle; and

the log data indicating the parameters of the autonomous vehicle over the second period of time comprises data from the GPS receiver of the autonomous vehicle.

14. A computing system comprising:

one or more processors; and

one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising;

comparing log data indicating positions of elements of a test track over a first period of time with log data indicating positions of the elements of the test track over a second period of time;

comparing log data indicating parameters of an autonomous vehicle over the first period of time with log data indicating the parameters of the autonomous vehicle over the second period of time; and

determining an effect of operations of the autonomous vehicle over a period of time between the first period of time and the second period of time, the effect caused by modifying software configured to control operations of the autonomous vehicle, wherein determining the effect comprises determining a difference in a measure of smoothness in transition from a first state to a second state of the autonomous vehicle.

15. The computing system of claim 14, wherein the elements of the test track comprise at least one of a simulated vehicle, a simulated pedestrian, or a simulated cyclist.

16. The computing system of claim 14, wherein the parameters of the autonomous vehicle comprise one or more of a rate of acceleration of the autonomous vehicle, a rate of deceleration of the autonomous vehicle, a speed of the autonomous vehicle, or a heading of the autonomous vehicle.

17. The computing system of claim 14, wherein determining the effect comprises determining the effect based at least in part on feedback, received from a user device, describing performance of the autonomous vehicle over the first period of time and the second period of time.

18. The computing system of claim 14, wherein:

the log data indicating the positions of the elements of the test track over the first period of time indicates a position or status of one or more actuators associated with the elements of the test track over the first period of time; and

the log data indicating the positions of the elements of the test track over the second period of time indicates a position or status of the one or more actuators associated with the elements of the test track over the second period of time.

19. The computing system of claim 14, wherein:

the log data indicating the positions of the elements of the test track over the first period of time comprises data from one or more global positioning system (GPS) receivers attached to the elements; and

the log data indicating the positions of the elements of the test track over the second period of time comprises data from the one or more GPS receivers attached to the elements.

20. The computing system of claim 14, wherein:

the log data indicating the parameters of the autonomous vehicle over the first period of time comprises data from a global positioning system (GPS) receiver of the autonomous vehicle; and

the log data indicating the parameters of the autonomous vehicle over the second period of time comprises data from the GPS receiver of the autonomous vehicle.