DRIVER GAZE DETECTION SYSTEM

• US 20140204193A1
• Filed: 09/30/2013
• Published: 07/24/2014
• Est. Priority Date: 01/18/2013
• Status: Active Grant

• Current Assignee: Carnegie Mellon University, GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: Carnegie Mellon University, GM Global Technology Operations LLC (General Motors Company)
• Inventors: Zhang, Wende, Nachtegall, Debbie E., Levi, Dan, De La Torre, Fernando, Huang, Dong
• Granted Patent: US 9,405,982 B2
• Time in Patent Office: Days
• US Class Current: 348/78
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

A method for detecting an eyes-off-the-road condition based on an estimated gaze direction of a driver of a vehicle includes monitoring facial feature points of the driver within image input data captured by an in-vehicle camera device. A location for each of a plurality of eye features for an eyeball of the driver is detected based on the monitored facial features. A head pose of the driver is estimated based on the monitored facial feature points. The gaze direction of the driver is estimated based on the detected location for each of the plurality of eye features and the estimated head pose.

Claims

1. Method for detecting an eyes-off-the-road condition based on an estimated gaze direction of a driver of a vehicle, comprising:

monitoring facial feature points of the driver within image input data captured by an in-vehicle camera device;

detecting a location for each of a plurality of eye features for an eyeball of the driver based on the monitored facial feature points;

estimating a head pose of the driver based on the monitored facial feature points; and

estimating the gaze direction of the driver based on the detected location information for each of the plurality of eye features and the estimated head pose.

2. The method of claim 1, wherein the monitored facial feature points are extracted in accordance with the steps, comprising:

for a first image frame, detecting a face of the driver and extracting the facial feature points from the detected face;

for each consecutive image frame subsequent to the first image frame;

identifying a candidate region encompassing the facial feature points extracted from one or more previous image input frames; and

identifying the facial feature points only within the candidate region.

3. The method of claim 1, wherein detecting the location for each of the plurality of eye features for the eyeball of the driver comprises:

detecting a location of an iris;

detecting a location for a first eye corner indicative of an inner eye corner proximate to a nose bridge; and

detecting a location for a second eye corner indicative of an outer eye corner.

4. The method of claim 3, wherein detecting the location for each of the plurality of eye features for the eyeball of the driver is further comprises:

for each corresponding eye feature of the plurality of eye features, training a classifier, comprising;

assigning image patches around a reference eye feature respective to the corresponding eye feature obtained from a database; and

identifying one or more of the assigned image patches that are centered around the reference eye feature.

5. The method of claim 4, further comprising:

for each corresponding eye feature within a first image frame;

selecting candidate pixels indicative of the corresponding eye feature using two statistical priors based on a magnitude of intensity and a detected edge strength;

calculating a confidence score for each candidate pixel based on a weighted sum of the two statistical priors;

selecting top candidate pixels each having a respective confidence score that is greater than a confidence score threshold;

comparing each of the top candidate pixels to the trained classifier to generate a classifier response respective to each of the top candidate pixels; and

detecting the location of the corresponding eye feature within the first image frame based on the generated classifier responses of the top candidate pixels.

6. The method of claim 5, further comprising:

for each corresponding eye feature within each consecutive image frame subsequent to the first image frame;

identifying supplemental facial feature points surrounding the corresponding eye feature detected in an immediately preceding image frame;

tracking a location change for each of the supplemental facial feature points from the immediately preceding image frame to the corresponding image frame;

estimating an initial location of the corresponding eye feature in the corresponding image frame based on the tracked location change of the supporting feature points;

calculating the confidence score for each of a plurality of pixels within an area surrounding the estimated initial location of the corresponding eye feature;

selecting top candidate pixels within the area surrounding the estimated initial location, wherein each top candidate pixel has a respective confidence score that is greater than the confidence score threshold;

comparing the top candidate pixels to the trained classifier to generate the classifier response for each of the top candidate pixels; and

detecting the location of the corresponding eye feature within the corresponding image frame based on the top candidate pixel having the highest classifier response.

7. The method of claim 6, wherein the plurality of pixels within the area surrounding the estimated initial location of the corresponding eye feature includes only pixels having detected edge responses exceeding a weak edge threshold.

8. The method of claim 1, wherein estimating the head pose of the driver based on the monitored facial feature points, comprises:

in an output space, generating a plurality of uniformly-spaced yaw angles within a range of yaw angles;

in an input space;

generating a plurality of subspaces each associated with respective ones of the uniformly-spaced yaw angles and parameterized by a respective mean and basis,

identifying two candidate subspaces among the plurality of subspaces closest to the monitored facial feature points, and

selecting a neighboring subspace among the two candidate subspaces having a lowest reconstruction error associated with the monitored facial feature points; and

estimating the head pose of the driver based on the uniformly-spaced yaw angle in the output space associated with the selected neighboring sub-space in the input space.

9. The method of claim 8, wherein estimating the head pose of the driver is further comprises:

sampling a plurality of training images in the input space each associated a respective trained yaw angle, wherein the trained yaw angles are non-uniformly-spaced within the range of angles in the output space;

reconstructing each trained image and each associated trained yaw angle based on the generated plurality of uniformly-spaced yaw angles and the generated plurality of subspaces; and

estimating the head pose of the driver based on one of the reconstructed training images and the trained yaw angle associated therewith.

10. The method of claim 1, further comprising:

prior to the monitoring facial feature points of the driver;

providing a detection region within the image input data encompassing a face location of the driver detected by a profile face detector;

using stored data of known face centers with respect to yaw angle, estimating a yaw angle of the detected face location of the driver within the detection region to generate a rectified face location; and

estimating the head pose of the driver within the image input data using the rectified face location.

11. The method of claim 1, wherein estimating the gaze direction of the driver based on the detected location information for each of the plurality of eye features and the estimated head pose, comprises:

detecting a location for each of first and second eye corners of the eyeball;

calculating a midpoint between the detected location for each of the first and second eye corners of the eyeball;

calculating a center of the eye ball using the calculated midpoint and two corrections based on the estimated head pose;

calculating a scale of the face of the driver based on a distance between the detected first and second eye corners and the estimated head pose;

calculating a radius of the eye ball based on multiplying a normalized radius of the eye ball with the calculated scale of the face; and

estimating the gaze direction of the driver based on the calculated radius of the eye ball and the calculated center of the eye ball.

12. The method of claim 1, further comprising:

identifying a gaze location of the driver corresponding to a point at which the estimated gaze direction intersects a windscreen plane of the vehicle;

comparing the identified gaze location of the driver to a road plane within the windscreen plane; and

detecting the eyes-off-the-road condition only when the identified gaze location is outside the road plane.

13. The method of claim 1, wherein the in-vehicle camera device comprises a monocular camera device mounted proximate to a steering wheel on an interior of the vehicle, the monocular camera device configured to capture a field of view directed toward the driver.

14. The method of claim 1, further comprising:

illuminating an interior of the vehicle using an infrared illuminator during low-light conditions within the interior of the vehicle.

15. The method of claim 1, further comprising:

prior to monitoring the facial feature points of the driver, performing an auto-calibration during a predetermined number of image frames such that the estimated head pose and the estimated gaze direction are adjusted specifically for the driver based on a generated bias, said auto-calibration comprising;

monitoring an inner ocular distance corresponding to a distance between a center of each eyeball of the driver;

determining a proportion of the head of the driver based on the monitored inner ocular distance;

calibrating the camera device to estimate a distance from the camera device to the driver;

calculating a displacement of a gaze of the driver from the camera device based on the estimated distance;

determining a height of the driver based on the displacement of gaze; and

generating the bias to adjust the estimated head pose and the estimated gaze direction of the driver based on the determined height and proportion of the head of the driver.

16. Apparatus for estimating gaze direction of a driver of a vehicle, comprising:

an in-vehicle monocular camera device for capturing image data in a field of view directed toward a face of the driver;

a detection and tracking module for extracting facial feature points of the driver from the image data captured by the camera device and detecting and tracking a location for each of an iris and first and second eye corners of an eyeball of the driver based on the extracted facial feature points;

a head pose estimation module utilizing the extracted facial feature points to estimate a head pose of the driver; and

a gaze direction estimation module for estimating the gaze direction of the driver in a geometric model based on the estimated head pose and the detected and tracked location for each of the iris and the first and second eye corners.

17. The apparatus of claim 16, wherein the detection and tracking module extracts the facial feature points from the face of the driver utilizing an Active Appearance Model (AAM), the face detected utilizing an Open Source Computer Vision face detector.

18. The apparatus of claim 16, wherein the detection and tracking module detects the location of each of the iris and first and second eye corners using a Support Vector Machine classifier and tracks the respective locations in consecutive image frames utilizing a Kanada-Lucas-Tomas method of tracking.

19. The apparatus of claim 16, wherein the head pose estimation module utilizes Supervised Local Subspace Learning to build local linear models from a sparse and non-uniformly spaced sampled training set to estimate the head pose of the driver.

20. The apparatus of claim 16, further comprising:

an infrared illuminator for illuminating an interior of the vehicle during low-light conditions.