System and method for predicting behaviors of detected objects through environment representation

• US 10,564,639 B1
• Filed: 08/07/2017
• Issued: 02/18/2020
• Est. Priority Date: 02/06/2012
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Zhu, Jiajun, Ferguson, David I.
• Primary Examiner(s): Nguyen, Cuong H
• Application Number:
  US15/670,128
• Time in Patent Office: 925 Days
• Field of Search: 701 23- 24, 701 26- 27, 701 311, 701300, 701439
• CPC Class Codes: B60W 30/0956 the prediction being responsive to traffic or environmental parameters
• CPC Class Codes: G01S 13/862 Combination of radar systems with sonar systems
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 15/931 of land vehicles
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 2013/932 using own vehicle data, e.g. ground speed, steering wheel direction
• CPC Class Codes: G01S 2013/9322 using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 2013/93271 in the front of the vehicles
• CPC Class Codes: G01S 2013/93272 in the back of the vehicles
• CPC Class Codes: G01S 2013/93273 on the top of the vehicles
• CPC Class Codes: G01S 2013/93274 on the side of the vehicles
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0257 using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G08G 1/096791 where the origin of the information is another vehicle
• CPC Class Codes: G08G 1/16 Anti-collision systems road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

Aspects of the invention relate generally to autonomous vehicles. The features described improve the safety, use, driver experience, and performance of these vehicles by performing a behavior analysis on mobile objects in the vicinity of an autonomous vehicle. Specifically, the autonomous vehicle is capable of detecting nearby objects, such as vehicles and pedestrians, and is able to determine how the detected vehicles and pedestrians perceive their surroundings. The autonomous vehicle may then use this information to safely maneuver around all nearby objects.

Claims

1. A method of controlling a vehicle in an autonomous driving mode, the method comprising:

controlling the vehicle in the autonomous driving mode based on a first control strategy;

detecting a set of objects in an external environment of the vehicle and a first location of each object of the set, the set of objects including two or more objects;

while controlling the vehicle based on the first control strategy, for each given object of the set of objects (1) determining a subset of other objects of the set of objects can be perceived by the given object and, (2) predicting a behavior for the given object based on the first locations of the subset of other objects;

adjusting the predicted behavior of a first object based on the predicted behavior of at least one other object of the set of objects;

determining a second control strategy by adjusting the first control strategy using the adjusted predicted behavior; and

controlling the vehicle in the autonomous driving mode based on the second control strategy.

2. The method of claim 1, further comprising, while controlling the vehicle according to the second control strategy:

updating the set of objects to include previously presented objects detected in the external environment, each object of the set of objects having second locations;

for each given object of the updated set of objects, determining an updated subset of other objects of the set of objects can be perceived by the given object and predicting a behavior for the given object of the updated set of objects based on the second locations of the updated subset of other objects;

adjust the predicted behavior of a second object based on the predicted behavior of at least one other object of the updated set of objects;

determine a third control strategy by adjusting the first control strategy using the adjusted predicted behavior of the second object; and

controlling the vehicle in the autonomous driving mode based on the third control strategy.

3. The method of claim 1, wherein the first object is a pedestrian.

4. The method of claim 3, wherein the at least one other object is a vehicle.

5. The method of claim 1, wherein the first object is a bicyclist.

6. The method of claim 1, wherein the first object is a first vehicle having a human driver.

7. The method of claim 6, wherein determining the subset of other objects is based on a line of sight of the human driver of the first vehicle.

8. The method of claim 6, wherein the at least one other object is a vehicle.

9. The method of claim 1, wherein the first object is an autonomous vehicle.

10. The method of claim 1, wherein adjusting the predicted behavior is performed after predetermined behaviors are determined for all of the objects of the set of objects.

11. The method of claim 10, further comprising adjusting the predicted behaviors of all objects of the set of objects based on the predicted behavior of one or more other objects of the set of objects.

12. The method of claim 1, further comprising for each given object of the set of objects, determining a relative location of each object of the subset of other objects relative to the given object, and wherein predicting the predicted behavior for the given object is based on the relative locations of each object of the subset of other objects.

13. A system for controlling a vehicle in an autonomous driving mode, the system comprising one or more processors configured to:

control the vehicle in the autonomous driving mode based on a first control strategy;

detect a set of objects in an external environment of the vehicle and a first location of each object of the set, the set of objects including two or more objects;

while controlling the vehicle based on the first control strategy, for each given object of the set of objects (1) determining a subset of other objects of the set of objects can be perceived by the given object and, (2) predicting a behavior for the given object based on the first locations of the subset of other objects;

adjust the predicted behavior of a first object based on the predicted behavior of at least one other object of the set of objects;

determine a second control strategy by adjusting the first control strategy using the adjusted predicted behavior; and

control the vehicle in the autonomous driving mode based on the second control strategy.

14. The system of claim 13, wherein the one or more processors are further configured to, while controlling the vehicle according to the second control strategy:

updating the set of objects to include previously presented objects detected in the external environment, each object of the set of objects having second locations;

for each given object of the updated set of objects, determine an updated subset of other objects of the set of objects can be perceived by the given object and predicting a behavior for the given object of the updated set of objects based on the second locations of the updated subset of other objects;

adjust the predicted behavior of a second object based on the predicted behavior of at least one other object of the updated set of objects;

determine a third control strategy by adjusting the first control strategy using the adjusted predicted behavior of the second object; and

control the vehicle in the autonomous driving mode based on the third control strategy.

15. The system of claim 13, wherein the first object is a first vehicle having a human driver.

16. The system of claim 15, wherein the one or more processors are further configured to determine the subset of other objects further based on a line of sight of the human driver of the first vehicle.

17. The system of claim 13, wherein the one or more processors are further configured to adjust the predicted behavior after predetermined behaviors are determined for all of the objects of the set of objects.

18. The system of claim 17, wherein the one or more processors are further configured to adjust the predicted behaviors of all objects of the set of objects based on the predicted behavior of one or more other objects of the set of objects.

19. The system of claim 13, wherein the one or more processors are further configured to, for each given object of the set of objects, determine a relative location of each object of the subset of other objects relative to the given object, and to predict the predicted behavior for the given object further based on the relative locations of each object of the subset of other objects.

20. The system of claim 13, further comprising the vehicle.