Predicting vehicle movements based on driver body language

• US 9,864,918 B2
• Filed: 11/04/2015
• Issued: 01/09/2018
• Est. Priority Date: 11/04/2015
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Micks, Ashley Elizabeth, Banvait, Harpreetsingh, Jain, Jinesh J, Reiff, Brielle
• Primary Examiner(s): STREGE, JOHN B
• Application Number:
  US14/932,160
• Publication Number:
  US 20170124407A1
• Time in Patent Office: 797 Days
• CPC Class Codes: B60W 40/08 related to drivers or passengers
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30268 Vehicle interior
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Systems, methods, and devices for predicting driver intent and future movements of a human driven vehicles are disclosed herein. A system for predicting future movements of a vehicle includes a camera system, a boundary component, a body language component, and a prediction component. The camera system is configured to capture an image of a vehicle. The boundary component is configured to identify a sub-portion of the image corresponding to an area where a driver of a vehicle is located. The body language component configured to detect a driver'"'"'s body language. The prediction component configured to predict future motion of the vehicle based on the driver'"'"'s body language detected by the body language component.

Claims

1. A system comprising:

a camera system configured to capture an image of a proximal vehicle;

a boundary component configured to identify a sub-portion of the image corresponding to an area where a driver of the proximal vehicle is located, wherein the boundary component identifies the sub-portion by identifying one or more windows of the proximal vehicle;

a body language component configured to detect a driver'"'"'s body language by processing data within the sub-portion of the image; and

a prediction component configured to predict future motion of the proximal vehicle based on the driver'"'"'s body language detected by the body language component;

wherein the prediction component is configured to access a database or model that correlates the driver'"'"'s body language detected by the body language component with one or more future vehicle movements;

wherein the database or model correlates a gesture with a current driving context, wherein the driving context comprises a stop at an intersection, an approach to an intersection, merging onto a roadway, exiting a roadway, entering a parking lot or parking spot, or exiting a parking lot or parking spot.

2. The system of claim 1, wherein the body language component is configured to detect a driver'"'"'s body language by identifying one or more of a driver'"'"'s head orientation, a gaze direction, and a gesture.

3. The system of claim 1, wherein the boundary component is configured to locate the proximal vehicle within the image.

4. The system of claim 1, wherein the boundary component is configured to identify the sub-portion of the image based on identification of a region of the vehicle where a driver would likely be located.

5. The system of claim 1, wherein the database or model correlates one or more of the following with the one or more future vehicle movements:

a waving motion of a hand;

a hand gesture comprising a palm facing toward a parent vehicle with fingers upward;

a gaze direction of the driver for a threshold period of time;

a series of head movements; and

a series of quick changes in gaze direction.

6. The system of claim 1, wherein the database or model correlates a gaze direction with a future driving direction.

7. A computer implemented method comprising:

receiving an image of a proximal vehicle in a region near a vehicle;

determining a region of the image that contains a driver of the proximal vehicle, wherein determining the region comprises determining based on a location of one or more windows of the proximal vehicle;

processing image data in the region of the image that contains the driver of the proximal vehicle to detect one or more of a head orientation, a gaze direction, and a gesture of the driver;

accessing a database or model that correlates one or more of the head orientation, the gaze direction, and the gesture with one or more future vehicle movements for the proximal vehicle; and

determining a vehicle maneuver for the vehicle based on the one or more future vehicle movements of the proximal vehicle;

wherein the database or model correlates a gesture with a current driving context, wherein the driving context comprises a stop at an intersection, an approach to an intersection, merging onto a roadway, exiting a roadway, entering a parking lot or parking spot, or exiting a parking lot or parking spot.

8. The method of claim 7, further comprising locating the proximal vehicle within the image.

9. The method of claim 7, wherein determining the region of the image that contains the driver comprises determining the region based on a predicted location of a driver'"'"'s seat in the vehicle.

10. The method of claim 7, wherein the database or model correlates one or more of the following with the one or more future vehicle movements:

a waving motion of a hand;

a hand gesture comprising a palm facing toward the vehicle with fingers upward;

a gaze direction of the driver for a threshold period of time;

a series of head movements; and

a series of changes in gaze direction.

11. Non-transitory computer readable storage media storing instructions that, when executed by one or more processors, cause the processors to:

receive an image of a proximal vehicle;

identify a boundary around a region of the image where a driver of the proximal vehicle is located, wherein the instructions cause the one or more processors to identify the boundary based on identifying one or more windows of the proximal vehicle;

detecting one or more of a driver'"'"'s head orientation, a gaze direction, and a gesture by processing image data in the boundary to detect the head orientation, the gaze direction, and the gesture; and

predict future motion of the proximal vehicle based on the driver'"'"'s body language by accessing a database or model that correlates the driver'"'"'s body language detected with one or more future vehicle movements;

wherein the database or model correlates a gesture with a current driving context, wherein the driving context comprises a stop at an intersection, an approach to an intersection, merging onto a roadway, exiting a roadway, entering a parking lot or parking spot, or exiting a parking lot or parking spot, wherein the driver'"'"'s body language comprise one or more of the driver'"'"'s head orientation, the gaze direction, and the gesture.

12. The computer readable storage media of claim 11, wherein the instructions further cause the processor to locate the proximal vehicle within the image.

13. The computer readable storage media of claim 11, wherein identifying the boundary around the region comprises identifying of a region of the vehicle where a driver would likely be located.

14. The computer readable storage media of claim 11, wherein the instructions further cause the processor to access a database or model that correlates one or more of the driver'"'"'s head orientation, the gaze direction, and the gesture with one or more future vehicle movements.

15. The computer readable storage media of claim 11, wherein the instructions further cause the one or more processor to determine a driving maneuver to be performed by a parent vehicle based on the predicted future motion.