Automated system and method for modeling the behavior of vehicles and other agents

• US 10,059,334 B1
• Filed: 09/28/2016
• Issued: 08/28/2018
• Est. Priority Date: 04/13/2012
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Zhu, Jiajun, Ferguson, David I., Dolgov, Dmitri A.
• Primary Examiner(s): Mahne, Kevin P
• Application Number:
  US15/278,338
• Time in Patent Office: 699 Days
• Field of Search: None
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2554/20 Static objects
• CPC Class Codes: B60W 2554/4023 large-size vehicles, e.g. trucks
• CPC Class Codes: B60W 2554/4029 Pedestrians
• CPC Class Codes: B60W 2554/80 Spatial relation or speed relative to objects
• CPC Class Codes: B60W 2555/20 Ambient conditions, e.g. wind or rain
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 30/08 Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 30/095 Predicting travel path or likelihood of collision
• CPC Class Codes: B60W 30/0956 the prediction being responsive to traffic or environmental parameters
• CPC Class Codes: B60W 50/0097 Predicting future conditions
• CPC Class Codes: B60W 60/00133 for resting
• CPC Class Codes: B60W 60/00274 considering possible movement changes
• CPC Class Codes: B60W 60/00276 for two or more other traffic participants
• CPC Class Codes: G01R 19/25 using digital measurement techniques
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0212 with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G05D 1/0214 : in accordance with safety or protection criteria, e.g. avoiding hazardous areas monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13
• CPC Class Codes: G05D 1/0289 : with means for avoiding collisions between vehicles vehicle fittings for automatically controlling speed including means for detecting potential obstacles B60K31/0008; avoiding obstacles by action on the steering system B62D; radar, sonar, lidar systems designed for anti-collision purposes G01S13/93, G01S15/93, G01S17/93
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06N 3/00 : Computing arrangements based on biological models
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/582 : of traffic signs
• CPC Class Codes: G06V 20/584 : of vehicle lights or traffic lights
• CPC Class Codes: G06V 20/588 : Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G08G 1/09623 : Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle G08G1/0967 takes precedence
• CPC Class Codes: G08G 1/09626 : where the origin of the information is within the own vehicle, e.g. a local storage device, digital map
• CPC Class Codes: G08G 1/096725 : where the received information generates an automatic action on the vehicle control
• CPC Class Codes: G08G 1/096741 : where the source of the transmitted information selects which information to transmit to each vehicle
• CPC Class Codes: G08G 1/096775 : where the origin of the information is a central station
• CPC Class Codes: G08G 1/164 : Centralised systems, e.g. external to vehicles
• CPC Class Codes: G08G 1/165 : for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

A method and apparatus are provided for determining one or more behavior models used by an autonomous vehicle to predict the behavior of detected objects. The autonomous vehicle may collect and record object behavior using one or more sensors. The autonomous vehicle may then communicate the recorded object behavior to a server operative to determine the behavior models. The server may determine the behavior models according to a given object classification, actions of interest performed by the object, and the object'"'"'s perceived surroundings.

Claims

1. A method of maneuvering a vehicle, the method comprising:

receiving, by one or more processors, object data, wherein the object data comprises a location of an object in an environment of the vehicle;

identifying, from the object data, a classification of the object;

identifying a behavior of interest corresponding to an action taken by the object based on the object data and on map information identifying a road element corresponding to the location of the object;

retrieving from memory, a behavior model for objects configured to predict how another object, having the same classification as the object, will behave based on the classification; and

maneuvering the vehicle, by the one or more processors, based on the behavior model and the identified behavior of interest,

wherein the object data further includes movement information for the object and identifying the behavior of interest is further based on a comparison between the road element and the movement information and the comparison indicates that the object has performed the behavior of interest.

2. The method of claim 1, wherein the road element comprise a first lane of traffic, and wherein the action taken indicates that the object has changed from a second lane of traffic to the first lane of traffic.

3. The method of claim 1, wherein the classification corresponds to a road vehicle.

4. The method of claim 1, wherein the classification corresponds to a pedestrian.

5. The method of claim 1, wherein the behavior model includes paths of travel for one or more previously monitored objects, and the method further comprises:

using the behavior model to identify a plurality of probabilities for potential actions of the object, and

wherein maneuvering the vehicle includes avoiding a trajectory for at least one of the potential actions.

6. The method of claim 5, further comprising updating the behavior model by adjusting the plurality of probabilities based on the behavior of interest.

7. The method of claim 1, further comprising creating, by the one or more processors, an initial behavior model for the object based on the object data and the map information, when a behavior model does not exist for a classification for the object.

8. The method of claim 1, wherein the behavior model provides information based on weather conditions.

9. A system for maneuvering a vehicle, the system comprising one or more processors configured to:

receive object data, wherein the object data comprises a location of an object in an environment of the vehicle;

identify, from the object data, a classification of the object;

identify a behavior of interest corresponding to an action taken by the object based on the object data and on map information identifying a road element corresponding to the location of the object;

retrieve from memory, a behavior model for objects configured to predict how another object, having the same classification as the object, will behave based on the classification; and

maneuver the vehicle based on the behavior model and the identified behavior of interest,

wherein the object data further includes movement information for the object, and the one or more processors are further configured to identify the behavior of interest further based on a comparison between the road element and the movement information and the comparison indicates that the object has performed the behavior of interest.

10. The system of claim 9, further comprising the vehicle.

11. The system of claim 9, further comprising the memory, and wherein the memory further stores the map information.

12. The system of claim 9, wherein the road element comprise a first lane of traffic, and wherein the action taken indicates that the object has changed from a second lane of traffic to the first lane of traffic.

13. The system of claim 9, wherein the classification corresponds to a road vehicle.

14. The system of claim 9, wherein the classification corresponds to a pedestrian.

15. The system of claim 9, wherein the behavior model includes paths of travel for one or more previously monitored objects, and the one or more processors are further configured to:

use the behavior model to identify a plurality of probabilities for potential actions of the object, and

to maneuver the vehicle by avoiding a trajectory for at least one of the potential actions.

16. The system of claim 15, wherein the one or more processors are further configured to update the behavior model by adjusting the plurality of probabilities based on the behavior of interest.

17. The system of claim 9, further comprising creating an initial behavior model for the object based on the object data and the map information, when a behavior model does not exist for a classification for the object.

18. The system of claim 9, wherein the behavior model provides information based on weather conditions.

20. The medium of claim 18, further comprising creating an initial behavior model for the object based on the object data and the map information, when a behavior model does not exist for a classification for the object.

19. A non-transitory, computer readable recording medium on which instructions are stored, the instructions, when executed by one or more processors, cause the one or more processors to perform a method of maneuvering a vehicle, the method comprising:

receiving object data, wherein the object data comprises a location of an object in an environment of the vehicle;

identifying, from the object data, a classification of the object;

identifying a behavior of interest corresponding to an action taken by the object based on the object data and on map information identifying a road element corresponding to the location of the object;

retrieving from memory, a behavior model for objects configured to predict how another object, having the same classification as the object, will behave based on the classification; and

maneuvering the vehicle based on the behavior model and the identified behavior of interest,

wherein the object data further includes movement information for the object and identifying the behavior of interest is further based on a comparison between the road element and the movement information and the comparison indicates that the object has performed the behavior of interest.