Method and system for estimating gaze direction of vehicle drivers

• US 9,881,221 B2
• Filed: 05/09/2014
• Issued: 01/30/2018
• Est. Priority Date: 10/24/2013
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Inventors: Bala, Raja, Bernal, Edgar, Burry, Aaron, Paul, Peter, Chuang, David
• Primary Examiner(s): Prince, Jessica M
• Application Number:
  US14/274,315
• Publication Number:
  US 20150116493A1
• Time in Patent Office: 1,362 Days
• Field of Search: 348148
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

Methods and systems for continuously monitoring the gaze direction of a driver of a vehicle over time. Video is received, which is captured by a camera associated with, for example, a mobile device within a vehicle, the camera and/or mobile device mounted facing the driver of the vehicle. Frames can then be extracted from the video. A facial region can then be detected, which corresponds to the face of the driver within the extracted frames. Features descriptors can then be computed from the facial region. A gaze classifier derived from the vehicle, the driver, and the camera can then be applied, wherein the gaze classifier receives the feature descriptors as inputs and outputs at least one label corresponding to one or more predefined finite number of gaze classes to identify the gaze direction of the driver of the vehicle.

Claims

1. A method for continuously monitoring the gaze direction of a driver of a vehicle over time, said method comprising:

receiving video captured by a mobile image-capturing unit comprising camera within a vehicle, said camera of said mobile image-capturing unit mounted facing a driver of said vehicle;

extracting frames from said video;

detecting a facial region corresponding to a face of said driver within said extracted frames;

computing feature descriptors from said facial region, said feature descriptors comprising at least one feature vector comprising a multi-dimension feature vector;

normalizing each component of said at least one feature vector Including face part locations and sizes associated with said facial region to render said feature descriptors invariant to facial translations and scaling as well as pixel resolution, wherein said normalizing comprises normalizing a position and size of said face by lengths of axes of a face coordinate system defined by a square surrounding said detected face with an origin located at a top-left corner of said square; and

applying a gaze classifier with respect to said vehicle, said driver, and said camera, wherein said gaze classifier receives said feature descriptors as inputs and outputs at least one label corresponding to at least one of a predefined finite number of common driving gaze directions to identify a gaze direction of said driver of said vehicle.

2. The method of claim 1 further comprising applying temporal filtering to said at least one label to obtain a smooth estimate of said gaze direction over time, such that transitions between different gazes are configured to occur smoothly relative to an acquisition frame rate of said frames extracted from said video.

3. The method of claim 2 further comprising training said gaze classifier by capturing in an offline operation, video segments of said driver corresponding to all gaze classes in said vehicle, while said vehicle is stationary.

4. The method of claim 3 further comprising training said gaze classifier by:

capturing in an offline operation video segments of said driver in said vehicle, while said vehicle is moving;

performing unsupervised clustering to extract training data corresponding to a road-facing class in said moving vehicle;

combining data from said stationary vehicle for all gaze classes with data from said moving vehicle for the road-facing class to form a training set; and

deriving said gaze classifier from said training set.

5. The method of claim 2 wherein detecting said facial region comprises running three detectors respectively for a frontal, left-, and right-facial profile, and wherein subsequent steps of computing feature descriptors and applying said gaze classifier are executed only for instances where a frontal face is detected.

6. The method of claim 5 further comprising deriving said three detectors for said frontal, left- and right-facial profile using Haar-like features and Adaboost cascade classifiers, wherein said unsupervised clustering is accomplished with an Expectation-Maximization (E-M) algorithm applied to learn a Gaussian mixture model (GMM) from available samples of said gaze direction.

7. The method of claim 6 further comprising follow-on actions depending on a desired application, said follow-on actions comprising providing to said driver real-time visual information, audio information, and a tactile notification that said driver is not being attentive.

8. A system for continuously monitoring the gaze direction of a driver of a vehicle over time, said system comprising:

a processor; and

a non-transitory computer-usable medium embodying computer program code, said non-transitory computer-usable medium capable of communicating with said processor, said computer program code comprising instructions executable by said processor and configured for;

receiving video captured by a mobile image-capturing unit comprising a camera within a vehicle, said camera of said mobile image-capturing unit mounted facing a driver of said vehicle;

extracting frames from said video;

detecting a facial region corresponding to a face of said driver within said extracted frames;

computing feature descriptors from said facial region, said feature descriptors comprising at least one feature vector comprising a multi-dimension feature vector;

normalizing each component of said at least one feature vector including face part locations and sizes associated with said facial region to render said feature descriptors invariant to facial translations and scaling as well as pixel resolution, wherein said normalizing comprises normalizing a position and size of said face by lengths of axes of a face coordinate system defined by a square surrounding said detected face with an origin located at a top-left corner of said square; and

applying a gaze classifier with respect to said vehicle, said driver, and said camera, wherein said gaze classifier receives said feature descriptors as inputs and outputs at least one label corresponding to at least one of a predefined finite number of gaze classes to identify a gaze direction of said driver of said vehicle.

9. The system of claim 8 wherein said instructions are further configured for applying temporal filtering to said at least one label to obtain a smooth estimate of said gaze direction over time, such that transitions between different gazes are configured to occur smoothly relative to the acquisition frame rate of said frames extracted from said video.

10. The system of claim 9 wherein said instructions are further configured for training said gaze classifier by capturing in an offline operation, video segments of said driver corresponding to all gaze classes in said vehicle, said vehicle comprising a stationary vehicle.

11. The system of claim 10 wherein said instructions are further configured for training said gaze classifier by:

capturing in an offline operation video segments of said driver in said vehicle, said vehicle comprising a moving vehicle;

performing unsupervised clustering to extract training data corresponding to a road-facing class in said moving vehicle;

combining data from said stationary vehicle for all gaze classes with data from said moving vehicle for the road-facing class to form a training set; and

deriving said gaze classifier from said training set.

12. The system of claim 10 wherein said instructions are further configured for detecting said facial region comprises running three detectors respectively for a frontal, left-, and right-facial profile, and wherein subsequent steps of computing feature descriptors and applying said gaze classifier are executed only for those instances where a frontal face is detected.

13. The system of claim 12 wherein said instructions are further configured for deriving said three detectors for said frontal, left-, and right-facial profile using Haar-like features and Adaboost cascade classifiers, wherein said unsupervised clustering is accomplished with an Expectation-Maximization (E-M) algorithm applied to learn a Gaussian mixture model (GMM) from available samples of said gaze direction.

14. The system of claim 13 wherein said instructions are further configured for conceiving follow-on actions depending on a desired application, said follow-on actions comprising providing to said driver:

real-time visual information, audio information, and a tactile notification that said driver is not being attentive.

15. A non-transitory processor-readable medium storing computer code representing instructions to cause a process for continuously monitoring the gaze direction of a driver of a vehicle over time, said computer code further comprising code to:

receive video captured by a mobile image-capturing unit comprising a camera within a vehicle, said camera of said mobile image-capturing unit mounted facing a driver of said vehicle;

extract frames from said video;

detect a facial region corresponding to a face of said driver within said extracted frames;

compute feature descriptors from said facial region, said feature descriptors comprising at least one feature vector comprising a multi-dimension feature vector;

normalize each component of said at least one feature vector including face part locations and sizes associated with said facial region to render said feature descriptors invariant to facial translations and scaling as well as pixel resolution, wherein said normalizing comprises normalizing a position and size of said face by lengths of axes of a face coordinate system defined by a square surrounding said defected face with an origin located at a top-left comer of said square; and

apply a gaze classifier with respect to said vehicle, said driver, and said camera, wherein said gaze classifier receives said feature descriptors as inputs and outputs at least one label corresponding to at least one of a predefined finite number of gaze classes to identify a gaze direction of said driver of said vehicle.

16. The processor-readable medium of claim 15 wherein said code further comprises code to apply temporal filtering to said at least one label to obtain a smooth estimate of said gaze direction over time, such that transitions between different gazes are configured to occur smoothly relative to the acquisition frame rate of said frames extracted from said video.

17. The processor-readable medium of claim 16 wherein said code further comprises code to train said gaze classifier by capturing in an offline operation, video segments of said driver corresponding to all gaze classes in said vehicle, said vehicle comprising a stationary vehicle.

18. The processor-readable medium of claim 17 wherein said code further includes code to train said gaze classifier by:

capturing in an offline operation video segments of said driver in said vehicle, said vehicle comprising a moving vehicle;

performing unsupervised clustering to extract training data corresponding to a road-facing class in said moving vehicle;

combining data from said stationary vehicle for all gaze classes with data from said moving vehicle for the road-facing class to form a training set; and

deriving said gaze classifier from said training set.

19. The processor-readable medium of claim 18 wherein said code further includes code to detect said facial region comprises running three detectors respectively for a frontal, left-, and right-facial profile, and wherein subsequent steps of computing feature descriptors and applying said gaze classifier are executed only for those instances where a frontal face is detected.

20. The processor-readable medium of claim 16 wherein said code further includes code to conceive follow-on actions depending on a desired application, said follow-on actions comprising providing to said driver:

real-time visual information, audio information, and a tactile notification that said driver is not being attentive.