Modeling behavior based on observations of objects observed in a driving environment

• US 9,120,484 B1
• Filed: 10/05/2011
• Issued: 09/01/2015
• Est. Priority Date: 10/05/2010
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ferguson, David I., Zhu, Jiajun, Dolgov, Dmitri A.
• Primary Examiner(s): Jabr, Fadey
• Assistant Examiner(s): Troost, Aaron L
• Application Number:
  US13/253,429
• Time in Patent Office: 1,427 Days
• Field of Search: 701/28, 701/223, 701/301, 382/104, 382/113, 382/169, 382/217, 706/46, 707/10
• US Class Current: 1/1
• CPC Class Codes: B60R 1/23 with a predetermined field of view
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60T 17/18 Safety devices; Monitoring
• CPC Class Codes: B60T 17/221 Procedure or apparatus for checking or keeping in a correct functioning condition of brake systems hydraulic pressure systems in general F15B19/00, F15B21/04; testing structures or apparatus G01M
• CPC Class Codes: B60T 2201/022 Collision avoidance systems
• CPC Class Codes: B60T 2210/32 Vehicle surroundings
• CPC Class Codes: B60T 2270/406 Test-mode; Self-diagnosis
• CPC Class Codes: B60T 7/22 initiated by contact of vehicle, e.g. bumper, with an external object, e.g. another vehicle , or by means of contactless obstacle detectors mounted on the vehicle
• CPC Class Codes: B60T 8/00 Arrangements for adjusting wheel-braking force to meet varying vehicular or ground-surface conditions, e.g. limiting or varying distribution of braking force by changing number of effective brake cylinders in power brake systems B60T17/10
• CPC Class Codes: B60T 8/17 Using electrical or electronic regulation means to control braking detecting or indicating faulty operation B60T8/885
• CPC Class Codes: B60T 8/885 using electrical circuitry
• CPC Class Codes: B60W 2050/0292 Fail-safe or redundant systems, e.g. limp-home or backup systems
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2552/05 Type of road, e.g. motorways, local streets, paved or unpaved roads
• CPC Class Codes: B60W 2555/60 Traffic rules, e.g. speed limits or right of way
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 2556/50 of positioning data, e.g. GPS [Global Positioning System] data
• CPC Class Codes: B60W 30/08 Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
• CPC Class Codes: B60W 30/186 excessive wear or burn out of friction elements, e.g. clutches
• CPC Class Codes: B60W 40/06 : Road conditions
• CPC Class Codes: B60W 50/14 : Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B62D 6/00 : Arrangements for automatically controlling steering depending on driving conditions sensed and responded to, e.g. control circuits means for initiating a change in direction B62D1/00; steering valves B62D5/06; combined with means for inclining the vehicle body or wheels on bends B62D9/00
• CPC Class Codes: G01C 21/3617 : using user history, behaviour, conditions or preferences, e.g. predicted or inferred from previous use or current movement
• CPC Class Codes: G01S 13/865 : Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 : Combination of radar systems with cameras
• CPC Class Codes: G01S 17/86 : Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 : of land vehicles
• CPC Class Codes: G05B 2219/2637 : Vehicle, car, auto, wheelchair
• CPC Class Codes: G05D 1/0055 : with safety arrangements
• CPC Class Codes: G05D 1/0088 : characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/021 : specially adapted to land vehicles
• CPC Class Codes: G05D 1/0212 : with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G05D 1/0214 : in accordance with safety or protection criteria, e.g. avoiding hazardous areas monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13
• CPC Class Codes: G05D 1/024 : in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0246 : using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0257 : using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G05D 1/0274 : using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0276 : using signals provided by a source external to the vehicle involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058
• CPC Class Codes: G05D 1/0278 : using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06Q 10/02 : Reservations, e.g. for tickets, services or events
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0207 : Discounts or incentives, e.g. coupons or rebates
• CPC Class Codes: G06Q 40/08 : Insurance
• CPC Class Codes: G06T 2207/10004 : Still image; Photographic image
• CPC Class Codes: G06T 2207/10028 : Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30236 : Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30252 : Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2207/30261 : Obstacle
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/223 : using block-matching
• CPC Class Codes: G06T 7/231 : using full search
• CPC Class Codes: G06T 7/521 : from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/74 : involving reference images or patches
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 : Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G07C 9/00563 : using personal physical data of the operator, e.g. finger prints, retinal images, voicepatterns

Abstract

A method and apparatus are provided for determining one or more object models used by an autonomous vehicle to predict the behavior of detected objects. The autonomous vehicle may collect and record object behavior using one or more sensors. The autonomous vehicle may then communicate the recorded object behavior to a server operative to determine the object models. The server may determine the object models according to a given object classification, a particular driving environment, or a combination thereof. The server may then communicate the object models to the autonomous vehicle for use in predicting the actions of detected objects and for responding accordingly.

Claims

1. A system for determining behavior data used by an autonomous vehicle, the system comprising:

a memory configured to store an object model; and

one or more processors in communication with the memory, the one or more processors configured to;

identify a type of driving environment and a geographic location of the driving environment;

receive, from a monitoring source, object data associated with a detected object;

analyze the object data to determine an object classification for the detected object, the object classification identifying a class of objects;

determine whether an object model currently exists for the object classification, wherein the object model correlates the expected behavior of objects within the determined object classification with the identified type of driving environment and the identified geographic location;

determine a new object model for the object classification based on the received object data when an object model does not currently exist for the object classification;

update the currently-existing object model for the object classification based on the received object data when an object model currently exists for the object classification; and

control the autonomous vehicle based on the object model.

2. The system of claim 1, wherein the one or more processors are further configured to identify the type of driving environment from data received from the monitoring source.

3. The system of claim 1, wherein the monitoring source is an autonomous vehicle.

4. The system of claim 1, wherein the detected object is a non-vehicle object.

5. The system of claim 1, wherein the object model comprises a plurality of probabilities, wherein at least one probability is based on a path of travel that the detected object was observed traveling.

6. The system of claim 5, wherein at least one probability of the plurality of probabilities identifies a probability that a detected object will travel a path of travel associated with the path of travel previously traveled by a previously detected object.

7. The system of claim 1, wherein the behavior data comprises a plurality of probabilities, wherein at least one probability is based on a speed at the detected object was observed moving.

8. The system of claim 7, wherein at least one probability of the plurality of probabilities identifies a probability that a detected object will travel at a speed associated with the speed at which a previously detected object was determined to be moving.

9. The system of claim 1, wherein the one or more processors are further configured to determine an object model for a plurality of types of driving environments.

10. The system of claim 1, wherein the one or more processors are further configured to communicate the object model to the autonomous vehicle.

11. The system of claim 1, wherein:

the monitoring source comprises the autonomous vehicle; and

the one or more processors are further configured to communicate a plurality of object models to the autonomous vehicle, wherein the one or more processors are remotely located from the autonomous vehicle.

12. A method for determining behavior data used by an autonomous vehicle, the method comprising:

identifying a type of driving environment and a geographic location of the driving environment;

receiving, with one or more processors, from a monitoring source, object data associated with a detected object;

analyzing, with the one or more processors, the object data to determine an object classification for the detected object, the object classification identifying a class of objects;

determining whether an object model currently exists for the expected behavior of objects within the determined object classification in the identified type of driving environment and at the identified geographic location;

determining a new object model for the object classification based on the received object data when an object model does not currently exist for the object classification;

updating the currently-existing object model for the object classification based on the received object data when an object model currently exists for the object classification; and

controlling the autonomous vehicle based on the object model.

13. The method of claim 12, further comprising identifying the driving environment from data received from the monitoring source.

14. The method of claim 12, wherein the monitoring source is an autonomous vehicle.

15. The method of claim 12, wherein the detected object is a non-vehicle object.

16. The method of claim 12, wherein the object model comprises a plurality of probabilities, wherein at least one probability is based on a path of travel that the detected object was observed traveling.

17. The method of claim 16, wherein at least one probability of the plurality of probabilities identifies a probability that a detected object will travel a path of travel associated with the path of travel previously traveled by a previously detected object.

18. The method of claim 12, wherein the behavior data comprises a plurality of probabilities, wherein at least one probability is based on a speed at which the detected object was observed moving.

19. The method of claim 18, wherein at least one probability of the plurality of probabilities identifies a probability that a detected object will travel at a speed associated with the speed at which a previously detected object was determined to be moving.

20. The method of claim 12, further comprising determining an object model for a plurality of types of driving environments.

21. The method of claim 12, further comprising communicating the object model to the autonomous vehicle.

22. The system of claim 12, wherein:

the monitoring source comprises the autonomous vehicle; and

the method further comprises communicating a plurality of object models to the autonomous vehicle, wherein at least one of the plurality of object models was determined remotely from the autonomous vehicle and at least one other object model was updated remotely from the autonomous vehicle.

23. A system for determining behavior data used by an autonomous vehicle, the apparatus system comprising:

a memory configured to store an object model; and

one or more processors in communication with the memory, the one or more processors configured to;

identify a type of driving environment and a geographic location of the driving environment;

receive object data associated with a plurality of detected objects from a source that monitored the behavior of the plurality of detected objects in the identified type of driving environment and the identified geographic location;

identify at least one object classification for each of the detected objects based on the received object data, each object classification identifying a class of objects

determine, for each identified object classification, whether an object model currently exists for the expected behavior of objects within the identified object classification in the identified type of driving environment and at the identified geographic location;

determine a new object model for an identified object classification based on the received object data when an object model does not currently exist for the object classification; and

update the currently-existing object model for an identified object classification based on the received object data when an object model currently exists for the object classification; and

control the autonomous vehicle based on the object model.