Simulating virtual objects

• US 9,836,895 B1
• Filed: 06/19/2015
• Issued: 12/05/2017
• Est. Priority Date: 06/19/2015
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Nygaard, Carl, Stout, James, Zhu, Jiajun
• Primary Examiner(s): Cass, Jean-Paul
• Application Number:
  US14/744,891
• Time in Patent Office: 900 Days
• Field of Search: 701 23
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06F 30/20 Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: G07C 5/0841 Registering performance data recording measured values G01D; information storage G11B

Abstract

An autonomous vehicle is tested using virtual objects. The autonomous vehicle is maneuvered, by one or more computing devices, the autonomous vehicle in an autonomous driving mode. Sensor data is received corresponding to objects in the autonomous vehicle'"'"'s environment, and virtual object data is received corresponding to a virtual object in the autonomous vehicle'"'"'s environment. The virtual object represents a real object that is not in the vehicle'"'"'s environment. The autonomous vehicle is maneuvered based on both the sensor data and the virtual object data. Information about the maneuvering of the vehicle based on both the sensor data and the virtual object data may be logged and analyzed.

Claims

1. A system for testing a vehicle using virtual objects, the system comprising:

one or more memories storing virtual object data, first behavior plan for a virtual object of the virtual object data, and alternative second behavior plan for the virtual object wherein the first behavior plan and the alternative second behavior plan each define different paths for the virtual object;

one or more testing computing devices in communication with the one or more memories, the one or more processors being configured to;

randomly select one of the first behavior plan or the alternative second behavior plan,

generate simulated sensor data for the virtual object based on the randomly selected one of the first behavior plan or the alternative behavior plan, and

send the simulated sensor data to one or more controlling computing devices, and

the one or more controlling computing devices in communication with the one or more memories, the one or more computing devices being configured to;

receive real sensor data corresponding to real objects in the vehicle'"'"'s environment detected by one or more sensors of the vehicle;

receive, from the one or more testing computing devices, the simulated sensor data corresponding to the virtual object being in the vehicle'"'"'s environment, such that the one or more controlling computing devices identify the virtual object as a real object in the vehicle'"'"'s environment that is executing the randomly selected one of the first behavior plan or the alternative second behavior plan even when the virtual object does not exist in the vehicle'"'"'s environment; and

maneuver the vehicle based on both the real object data and the simulated sensor data to test a maneuvering response of the vehicle.

2. The system of claim 1, wherein the one or more testing computing device is further configured to receive, from the one or more controlling computing devices, information about the maneuvering of the vehicle based on both the sensor data and the virtual simulated sensor data and record, in the one or more memories, the information.

3. The system of claim 2, the one or more testing computing devices are further configured to analyze the recorded information to evaluate the maneuvering response.

4. The system of claim 1, wherein the randomly selected one of the first behavior plan or the alternative second behavior plan identifies a path of locations the virtual object at two or more times.

5. The system of claim 4, wherein the randomly selected one of the first behavior plan or the alternative second behavior plan identifies at least one location of the virtual object, a direction of movement of the virtual object, a speed of the virtual object, and at least one physical characteristic of the virtual object.

6. The system of claim 4, wherein the one or more testing computing devices are further configured to log information about the maneuvering of the vehicle at the two or more times.

7. The system of claim 6, wherein the logging includes recording information about whether the vehicle experienced an error at the two or more times.

8. The system of claim 4, wherein the one or more testing computing devices are further configured to determine based on the path of locations whether the vehicle crossed a given location of the virtual object at a time when the virtual object was at the given location.

9. The system of claim 1, wherein receiving the simulated sensor data is triggered by a status of the vehicle.

10. The system of claim 1, further comprising the vehicle.

11. A method for testing a vehicle using virtual objects, the method comprising:

randomly selecting, by one or more testing computing devices, one of a first behavior plan for a virtual object or alternative second behavior plan for the virtual object, wherein the first behavior plan and the alternative second behavior plan each define different paths for the virtual object,

generating, by the one or more testing computing devices, simulated sensor data for the virtual object based on the randomly selected one of the first behavior plan or the alternative behavior plan, and

sending, by the one or more testing computing devices, the simulated sensor data to one or more controlling computing devices, and

receiving, by one or more controlling computing devices, real sensor data corresponding to real objects in the vehicle'"'"'s environment detected by one or more sensors of the vehicle;

receiving, from the one or more testing computing devices by the one or more controlling computing devices, the simulated sensor data corresponding to the virtual object being in the vehicle'"'"'s environment, such that the one or more controlling computing devices identify the virtual object as a real object in the vehicle'"'"'s environment that is executing the randomly selected one of the first behavior plan or the alternative second behavior plan even when the virtual object does not exist in the vehicle'"'"'s environment; and

maneuvering, by the one or more controlling computing devices, the vehicle based on both the real object data and the simulated sensor data to test a maneuvering response of the vehicle.

12. The method of claim 11, further comprising logging, by the one or more testing computing devices, information about the maneuvering of the vehicle based on both the sensor data and the virtual object data for testing a maneuvering response of the vehicle.

13. The method of claim 11, further comprising, prior to randomly selecting one of the first behavior plan or the alternative second behavior plan, receiving from a user in the vehicle a user selection identifying both the first behavior plan and the alternative second behavior plan.

14. The method of claim 12, further comprising receiving behavior information for the virtual object, the behavior information identifying a path of locations the virtual object at two or more times.

15. The method of claim 14, wherein the logging includes recording the information about the maneuvering of the vehicle at the two or more times.

16. The method of claim 14, further comprising:

determining based on the path of locations whether the vehicle crossed a given location of the virtual object at a time when the virtual object was at the given location, and

wherein the logging includes recording information about whether the vehicle crossed the given location of the virtual object.

17. The method of claim 12, further comprising reviewing, by the one or more testing computing devices, the logged information to evaluate the maneuvering response.

18. The method of claim 14, wherein the logging includes recording information about whether the vehicle experienced an error at the two or more times.

19. The method of claim 11, wherein the randomly selected one of the first behavior plan or the alternative second behavior plan for the virtual object identifies at least one location of the virtual object, a direction of movement of the virtual object, a speed of the virtual object, and at least one physical characteristic of the virtual object.

20. The method of claim 11, wherein the simulated sensor data further includes information about a fictitious state of an actual traffic signal light.