System and method for predicting behaviors of detected objects

• US 8,660,734 B2
• Filed: 10/03/2011
• Issued: 02/25/2014
• Est. Priority Date: 10/05/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Zhu, Jiajun, Ferguson, David I., Dolgov, Dmitri A.
• Primary Examiner(s): Tarcza, Thomas
• Assistant Examiner(s): Dunn, Alex C
• Application Number:
  US13/251,705
• Publication Number:
  US 20120083960A1
• Time in Patent Office: 876 Days
• Field of Search: 701/23
• US Class Current: 701/23
• CPC Class Codes: B60R 1/23 with a predetermined field of view
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60T 17/18 Safety devices; Monitoring
• CPC Class Codes: B60T 17/221 Procedure or apparatus for checking or keeping in a correct functioning condition of brake systems hydraulic pressure systems in general F15B19/00, F15B21/04; testing structures or apparatus G01M
• CPC Class Codes: B60T 2201/022 Collision avoidance systems
• CPC Class Codes: B60T 2210/32 Vehicle surroundings
• CPC Class Codes: B60T 2270/406 Test-mode; Self-diagnosis
• CPC Class Codes: B60T 7/22 initiated by contact of vehicle, e.g. bumper, with an external object, e.g. another vehicle , or by means of contactless obstacle detectors mounted on the vehicle
• CPC Class Codes: B60T 8/00 Arrangements for adjusting wheel-braking force to meet varying vehicular or ground-surface conditions, e.g. limiting or varying distribution of braking force by changing number of effective brake cylinders in power brake systems B60T17/10
• CPC Class Codes: B60T 8/17 Using electrical or electronic regulation means to control braking detecting or indicating faulty operation B60T8/885
• CPC Class Codes: B60T 8/885 using electrical circuitry
• CPC Class Codes: B60W 2050/0292 Fail-safe or redundant systems, e.g. limp-home or backup systems
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2552/05 Type of road, e.g. motorways, local streets, paved or unpaved roads
• CPC Class Codes: B60W 2555/60 Traffic rules, e.g. speed limits or right of way
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 2556/50 of positioning data, e.g. GPS [Global Positioning System] data
• CPC Class Codes: B60W 30/08 Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
• CPC Class Codes: B60W 30/186 excessive wear or burn out of friction elements, e.g. clutches
• CPC Class Codes: B60W 40/06 : Road conditions
• CPC Class Codes: B60W 50/14 : Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B62D 6/00 : Arrangements for automatically controlling steering depending on driving conditions sensed and responded to, e.g. control circuits means for initiating a change in direction B62D1/00; steering valves B62D5/06; combined with means for inclining the vehicle body or wheels on bends B62D9/00
• CPC Class Codes: G01C 21/3617 : using user history, behaviour, conditions or preferences, e.g. predicted or inferred from previous use or current movement
• CPC Class Codes: G01S 13/865 : Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 : Combination of radar systems with cameras
• CPC Class Codes: G01S 17/86 : Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/931 : of land vehicles
• CPC Class Codes: G05B 2219/2637 : Vehicle, car, auto, wheelchair
• CPC Class Codes: G05D 1/0055 : with safety arrangements
• CPC Class Codes: G05D 1/0088 : characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/021 : specially adapted to land vehicles
• CPC Class Codes: G05D 1/0212 : with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G05D 1/0214 : in accordance with safety or protection criteria, e.g. avoiding hazardous areas monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13
• CPC Class Codes: G05D 1/024 : in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0246 : using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0257 : using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G05D 1/0274 : using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0276 : using signals provided by a source external to the vehicle involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058
• CPC Class Codes: G05D 1/0278 : using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06Q 10/02 : Reservations, e.g. for tickets, services or events
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0207 : Discounts or incentives, e.g. coupons or rebates
• CPC Class Codes: G06Q 40/08 : Insurance
• CPC Class Codes: G06T 2207/10004 : Still image; Photographic image
• CPC Class Codes: G06T 2207/10028 : Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30236 : Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30252 : Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2207/30261 : Obstacle
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/223 : using block-matching
• CPC Class Codes: G06T 7/231 : using full search
• CPC Class Codes: G06T 7/521 : from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/74 : involving reference images or patches
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 : Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G07C 9/00563 : using personal physical data of the operator, e.g. finger prints, retinal images, voicepatterns

Abstract

Aspects of the invention relate generally to autonomous vehicles. Specifically, the features described may be used alone or in combination in order to improve the safety, use, driver experience, and performance of these vehicles.

Claims

1. A method comprising:

controlling a vehicle according to a first control strategy, the first control strategy determined based on a route to a destination;

detecting an object external to the vehicle using one or more sensors;

using a processor, determining a classification and a state of the detected object;

determining a current location of the vehicle;

accessing behavior data for other objects having the same or similar classifications and states as the detected object, the behavior data indicating how the other objects have operated at the current location at one or more previous times;

predicting a likely behavior of the detected object at the current location based on the behavior data for the other objects having the same or similar classification and state as the detected object; and

modifying the first control strategy to obtain a second control strategy for controlling the vehicle based on the predicted likely behavior of the detected object.

2. The method of claim 1, wherein the classification of the detected object includes one of an automobile, a pedestrian, structure, or a bicycle.

3. The method of claim 1, wherein the detected object is an automobile, and wherein the classification of the detected object includes the type of automobile.

4. The method of claim 1, wherein the determining a classification of the detected object is based on identifying at least one of a logo, a bumper sticker, or a license plate.

5. The method of claim 1, wherein the state of the detected object relates to at least one of:

location, traffic lane in which the detected object is traveling, speed, acceleration, entry onto a road, exit off of a road, activation of headlights, activation of taillights, or activation of blinkers.

6. The method of claim 1, wherein the behavior data is based on movement data for a plurality of other objects at one or more locations.

7. The method of claim 6, wherein the movement data are tracked using one of:

satellite imagery, roadside cameras, on-board GPS data, or sensor data acquired for other nearby vehicles.

8. The method of claim 1, wherein:

implementing the second control strategy comprises providing a command to orient the vehicle in a position and velocity based at least in part on the likely behavior of the detected object; and

providing the command to orient the vehicle includes positioning the vehicle at a predetermined distance from the detected object, the predetermined distance being based, at least in part, on the classification of the detected object.

9. The method of claim 1, wherein the likely behavior of the detected object is provided as a probability of the detected object entering to one or more states.

10. The method of claim 1, further comprising:

receiving updated behavior data; and

wherein predicting the likely behavior of the detected object is based at least in part on the updated behavior data.

11. The method of claim 10, wherein receiving the updated behavior data comprises accessing a remote server.

12. A system comprising:

one or more sensors for detecting an object in a vehicle'"'"'s surroundings; and

a processor coupled to the one or more sensors, the processor being operable to;

control operation of the vehicle based on a first control strategy, wherein the first control strategy is determined based on a route to a destination;

determine a classification and a state of a detected object;

determine a current location of the vehicle;

access behavior data for other objects having the same or similar classifications and states similar as the detected object, the behavior data indicating how the other objects have operated at the current location at one or more previous times;

predict a likely behavior of the detected object at the current location based on the behavior data for the other objects having the same or similar classification and state as the detected object; and

modify the first control strategy to obtain a second control strategy for controlling the vehicle based on the predicted likely behavior of the detected object.

13. The system of claim 12, wherein the classification of the detected object includes classifying the detected object as one of:

an automobile, a pedestrian, a bicycle, or a structure.

14. The system of claim 12, wherein the detected object is an automobile, and wherein the classification of the detected object includes identifying the type of automobile.

15. The system of claim 12, wherein the classification of the detected object is based on at least one of:

a logo, a bumper sticker, or a license plate.

16. The system of claim 12, wherein the state of the detected object relates to at least one of:

location, traffic lane in which the detected object is traveling, speed, acceleration, entry onto a road, exit off of a road, activation of headlights, activation of taillights, or activation of blinkers.

17. The system of claim 12, wherein the behavior data represents information regarding the states of the other objects that have been tracked at one or more locations.

18. The system of claim 17, wherein the objects have been tracked using at least one of:

satellite imagery, roadside cameras, on-board GPS data, or via sensor data acquired from other nearby entities.

19. The system of claim 12, wherein the second control strategy includes positioning the vehicle at a predetermined distance from the detected object, the predetermined distance being based, at least in part, on the classification of the detected object.

20. The system of claim 12, wherein the likely behavior of the detected object is provided as a probability of the detected object entering to one or more states.

21. The system of claim 12,

wherein the processor is further operable to access updated behavior data;

and

wherein the likely behavior of the detected object is based at least in part on the updated behavior data.

22. The system of claim 21, wherein the vehicle receives the updated behavior data from a remote server.

23. A method for providing data in connection with controlling of a vehicle, the method comprising:

storing, at a server, behavior data for a plurality of objects, wherein the behavior data includes classification and state information for at least a subset of the plurality of objects at one or more locations and at one or more times;

receiving identification data from a remote device, wherein the identification data includes data relating to a detected object near the remote device;

receiving a location of the detected object;

determining, at the server, the classification and state of the detected object;

predicting a likely behavior of the detected object based on the behavior data for the plurality of objects having a classification and state similar to the detected object for the location where the detected object was detected and the time when the detected object was detected;

transmitting the likely behavior of the detected object to the remote device.