Autonomous driving at intersections based on perception data

• US 9,983,591 B2
• Filed: 11/05/2015
• Issued: 05/29/2018
• Est. Priority Date: 11/05/2015
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Micks, Ashley Elizabeth, Banvait, Harpreetsingh, Jain, Jinesh J, Reiff, Brielle
• Primary Examiner(s): Berona, Kimberly
• Assistant Examiner(s): Malhotra, Sanjeev
• Application Number:
  US14/933,693
• Publication Number:
  US 20170131719A1
• Time in Patent Office: 936 Days
• Field of Search: 701 36, 701 41, 701117, 701300, 701301, 3404255, 340436, 340438, 3406916, 340905, 340936, 348116, 348148, 382104, 345 76, 345427, 359603
• CPC Class Codes: B60W 2520/10 Longitudinal speed
• CPC Class Codes: B60W 2520/105 Longitudinal acceleration
• CPC Class Codes: B60W 2554/4041 Position
• CPC Class Codes: B60W 2554/4042 Longitudinal speed
• CPC Class Codes: B60W 2554/4045 Intention, e.g. lane change or imminent movement
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 30/095 Predicting travel path or likelihood of collision
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: B60W 50/0097 Predicting future conditions
• CPC Class Codes: B60W 60/0011 involving control alternatives for a single driving scenario, e.g. planning several paths to avoid obstacles
• CPC Class Codes: B60W 60/0015 specially adapted for safety
• CPC Class Codes: B60W 60/0027 using trajectory prediction for other traffic participants
• CPC Class Codes: B60W 60/00274 considering possible movement changes
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0257 using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/584 : of vehicle lights or traffic lights
• CPC Class Codes: G06V 40/161 : Detection; Localisation; Normalisation
• CPC Class Codes: G08G 1/161 : Decentralised systems, e.g. inter-vehicle communication
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

Systems, methods, and devices for predicting a driver'"'"'s intention and future movements of a proximal vehicle, whether an automated vehicle or a human driven vehicle, are disclosed herein. A system for predicting future movements of a vehicle includes an intersection component, a camera system, a boundary component, and a prediction component. The intersection component is configured to determine that a parent vehicle is near an intersection. The camera system is configured to capture an image of the proximal vehicle. The boundary component is configured to identify a sub-portion of the image containing a turn signal indicator on the proximal vehicle. The prediction component is configured to predict future movement of the proximal vehicle through the intersection based on a state of the turn signal indicator.

Claims

1. A system comprising:

an intersection component configured to determine that a parent vehicle is near an intersection;

a camera system configured to capture an image of a proximal vehicle;

a boundary component configured to identify a sub-portion of the image containing a turn signal indicator on the proximal vehicle and a driver of the proximal vehicle;

a turn signal component configured to process image data in the sub-portion of the image to determine the state of the turn signal indicator;

a prediction component configured to predict future movement of the proximal vehicle through the intersection based on a state of the turn signal indicator; and

at least one or more processors configured to;

process image data in the sub-portion of the image containing the driver of the proximal vehicle to determine a body language of the driver;

predict future movement of the proximal vehicle through the intersection based on the state of the turn signal indicator and the body language of the driver;

determine a time for the parent vehicle to proceed through the intersection based on a predicted future movement of the proximal vehicle; and

wherein the at least one or more processors or an actuator causes the parent vehicle to perform a driving maneuver based on a determined time to proceed through the intersection.

2. The system of claim 1, further comprising a previous state component configured to determine one or more previous states of the proximal vehicle based on wireless communications indicating the one or more previous states of the proximal vehicle, wherein the prediction component is configured to predict future movements of the proximal vehicle based on the one or more previous states of the proximal vehicle.

3. The system of claim 2, wherein the wireless communication comprises one or more of a vehicle-to-vehicle (V2V) communication and a vehicle-to-infrastructure (V2X) communication.

4. The system of claim 2, wherein the one or more previous states indicate a duration of time during which the proximal vehicle has been located near the intersection.

5. The system of claim 1, further comprising a vehicle movement component configured to determine one or more vehicle movements of the proximal vehicle, wherein the prediction component is configured to predict future movements of the proximal vehicle based on the one or more vehicle movements of the proximal vehicle.

6. The system of claim 1, wherein the boundary component is further configured to identify a sub-portion of the image corresponding to a location of a driver, the system further comprising a body language component configured to detect a driver'"'"'s body language by identifying one or more of a driver'"'"'s head orientation, and a gaze direction, wherein the prediction component is configured to predict future movements of the proximal vehicle based on the driver'"'"'s body language.

7. A computer implemented method comprising:

receiving an image of a proximal vehicle near an intersection and storing the image in computer memory;

identifying, using one or more processors, a sub-portion of the image containing a turn signal indicator on the proximal vehicle;

identifying, using the one or more processors, a sub-portion of the image containing a driver of the proximal vehicle;

processing, using the one or more processors, image data in the sub-portion of the image containing a turn signal indicator to determine the state of the turn signal indicator;

processing, using the one or more processors, image data in the sub-portion of the image containing a driver of the proximal vehicle to determine a body language of the driver;

predicting, using the one or more processors, future movement of the proximal vehicle through the intersection based on the state of the turn signal indicator and the body language of the driver;

determining, using the one or more processors, a time for a parent vehicle to proceed through the intersection based on a predicted future movement of the proximal vehicle; and

causing, using one or more processors or actuators, the parent vehicle to perform a driving maneuver based on a determined time to proceed through the intersection.

8. The method of claim 7, further comprising determining that the parent vehicle is near an intersection.

9. The method of claim 7, wherein determining the body language of the driver comprises identifying one or more of a driver'"'"'s head orientation, a gaze direction, and a gesture.

10. The method of claim 7, further comprising determining one or more previous states of the proximal vehicle based on a wireless communication, wherein predicting future movements of the proximal vehicle comprises predicting based on the one or more previous states of the proximal vehicle.

11. The method of claim 10, wherein the wireless communication comprises one or more of a vehicle-to-vehicle (V2V) communication and a vehicle-to-infrastructure (V2X) communication.

12. The method of claim 7, further comprising detecting one or more vehicle movements of the proximal vehicle, wherein predicting future movements of the proximal vehicle comprises predicting based on the one or more vehicle movements of the proximal vehicle.

13. The method of claim 7, further comprising accessing or processing a model or database correlating the state of the turn signal indicator and the body language of the driver with the predicted future movement.

14. The method of claim 13, wherein the model or database comprises machine learning values or correlations based on motion of one or more vehicles, driver body language, and turn signal information during previous intersection driving scenarios.

15. Computer readable non-transitory storage media storing instructions that, when executed by one or more processors, cause the processors to:

determine that a parent vehicle is near an intersection;

capture a plurality of images of a proximal vehicle;

identify a sub-portion in each of the plurality of the images containing a turn signal indicator on the proximal vehicle;

process image data in the sub-portion of the plurality of images to determine the state of the turn signal indicator;

predict future movement of the proximal vehicle through the intersection based on a state of the turn signal indicator;

process, using the one or more processors, image data in the sub-portion of the image containing a driver of the proximal vehicle to determine a body language of the driver;

predict, using the one or more processors, future movement of the proximal vehicle through the intersection based on the state of the turn signal indicator and the body language of the driver;

determine, using the one or more processors, a time for a parent vehicle to proceed through the intersection based on a predicted future movement of the proximal vehicle; and

cause, using one or more processors or actuators, the parent vehicle to perform a driving maneuver based on a determined time to proceed through the intersection.

16. The computer readable storage media of claim 15, wherein the instructions further cause the processor to:

determine one or more previous states of the proximal vehicle based on wireless communications indicating the one or more previous states of the proximal vehicle; and

predict future movements of the proximal vehicle based on the one or more previous states of the proximal vehicle.

17. The computer readable storage media of claim 15, wherein the instructions further cause the processor to:

identify a sub-portion of the image corresponding to a location of a driver;

detect a driver'"'"'s body language by identifying one or more of a driver'"'"'s head orientation, a gaze direction, and a gesture; and

predict future movements of the proximal vehicle based on the driver'"'"'s body language.