Driver distraction and drowsiness warning and sleepiness reduction for accident avoidance

• US 9,460,601 B2
• Filed: 01/05/2014
• Issued: 10/04/2016
• Est. Priority Date: 09/20/2009
• Status: Active Grant

• Current Assignee: Tibet Mimar
• Original Assignee: Tibet Mimar
• Inventors: Mimar, Tibet
• Primary Examiner(s): SENFI, BEHROOZ M
• Application Number:
  US14/147,580
• Publication Number:
  US 20140139655A1
• Time in Patent Office: 1,003 Days
• Field of Search: 340/575, 348/148, 348/77
• US Class Current: 1/1
• CPC Class Codes: G08B 21/0476 Cameras to detect unsafe condition, e.g. video cameras
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms

Abstract

The present invention relates to a vehicle telematics device for driver monitoring for accident avoidance for drowsiness and distraction conditions. The distraction and drowsiness is detected by facial processing of driver'"'"'s face and pose tracking as a function of speed and maximum allowed travel distance, and issuing a driver alert when a drowsiness or distraction condition is detected. The mitigation includes audible alert, as well as other methods such as dim blue night to perk up the driver. Adaptation center of driver'"'"'s gaze direction and allowed maximum time for a given driver and camera angle offset as well as temporary offset for cornering for shift of vanishing point and other conditions is also performed.

Claims

1. A method for a driver drowsiness warning and accident avoidance system for a vehicle, comprising the steps of:

a) determining speed of said vehicle;

b) calculating a maximum allowed drowsiness time in accordance with speed of said vehicle and allowed drowsiness travel distance;

wherein said maximum allowed drowsiness time is a non-linear function of said speed of said vehicle;

c) determining a score of confidence of detecting driver'"'"'s face and facial features;

d) determining driver'"'"'s level of the driver'"'"'s left eye closed and the driver'"'"'s right eye closed, if said score is larger than a first threshold value;

e) calculating level of eyes closed as maximum of said driver'"'"'s left eye closed level and said driver'"'"'s right eye closed level;

f) filtering said calculated level of eyes closed;

g) issuing a driver drowsiness alarm, if said filtered calculated level of eyes closed exceed a second threshold value persist longer than said maximum allowed drowsiness time; and

h) illuminating the driver'"'"'s face with approximately 460 nm dim blue light to increase alertness of driver when said driver drowsiness alarm is issued at night time.

2. The method claim of 1, further comprising the steps of:

a) determining the driver'"'"'s face gaze direction;

b) if the driver'"'"'s face gaze direction has a roll angle or a tilt angle that exceeds a third threshold value;

c) determining if condition of (b) persists more than a time duration of fourth threshold value, wherein the fourth threshold value can be the same as said maximum allowed drowsiness time or a different value; and

d) issuing said driver drowsiness alarm if condition of (c) is true even if level of eyes closed cannot be determined due to occlusion.

3. The method claim of 1, wherein driver'"'"'s face gaze direction is determined using one of method including but not limited to active appearance model, cylinder-head model, appearance template method, flexible models with active appearance models, geometric methods for facial features, tracking methods for feature tracking using affine transformation and appearance-based particle filters, and hybrid methods that includes one and more methods combined from a list of geometric method and tracking, appearance template and tracking, active appearance models, and cylinder-head models.

4. The method claim of 1, further comprising the steps of:

illuminating the driver'"'"'s face by one of methods including but not limited to dim visible light and infrared light that is not visible to a human when ambient light level is low,

wherein a camera lens system supports a near infrared light bandpass when infrared light is used for illumination in accordance with ambient light conditions.

5. The method claim of 1, further comprising the steps of:

a) detecting the area of facial coordinates of the driver;

b) adding a padding area around the said area of facial coordinates of the driver;

c) performing auto-exposure algorithm weighted inside said padding area; and

d) updating said detected area continuously in accordance with the video stream of frames of the driver'"'"'s face.

6. The method claim of 1, further comprising the step of:

using other mitigation methods when drowsiness is detected further including but not limited to vibrating driver'"'"'s seat, multiple levels of said blue light for perking up the driver, turning on the vehicle'"'"'s emergency flashers, automatically calling a friend, and lowering the temperature of inside said vehicle.

7. The method claim of 1, further comprising the steps of:

connecting to internet when said driver drowsiness warning is issued; and

communicating drowsiness condition to a pre-determined destination which includes but not limited to one or more of fleet management for driver analytics, parent(s), highway patrol, insurance company for driver analytics, and family and friends.

8. A method for a driver distraction warning system for a vehicle for accident avoidance and driver analytics, comprising the steps of:

a) capturing images of the driver'"'"'s face region using a high-dynamic range(HDR) image sensor under varying illumination conditions;

b) removing noise components using MATF and MASF filtering from said captured images;

c) determining a current speed of the vehicle, and using a past average speed value if said current speed cannot be determined;

d) calculating a maximum allowed distraction time in accordance with a maximum allowed distracted travel distance,

wherein the maximum allowed distraction time is a non-linear function of said maximum allowed distracted travel distance;

e) determining a score of confidence of detecting the driver'"'"'s face and facial features from said filtered captured images;

f) determining the driver'"'"'s face gaze direction, if said score is larger than a predetermined score threshold;

g) filtering said driver'"'"'s face gaze direction values over multiple frames of said filtered captured images;

h) determining if the driver'"'"'s filtered face gaze direction is outside a non-distraction window of view;

i) calculating a time duration when the driver'"'"'s filtered face gaze direction stays outside the non-distraction window; and

j) issuing an at least one alert warning to the driver when the time duration of filtered face gaze direction exceeds a time threshold value if the current speed of the vehicle is larger than a low speed threshold value.

9. The method claim of 8, wherein said at least one alert warning includes but not limited to one of methods of sound or chime warning, turning on emergency flashers, limiting the speed of the vehicle to minimum allowed speed, and the driver'"'"'s seat vibration.

10. The method claim of 8, further comprising the steps of:

a) capturing images of the driver'"'"'s face region using a second high-dynamic range(HDR) image sensor;

b) determining a second face gaze direction value and a second confidence score using said second HDR image sensor input; and

c) merging multiple face gaze direction and confidence score values.

11. The method claim of 8, further comprising the step of:

a) determining the x-y-z gyro sensor inputs in accordance with curvature of road condition to tangent point;

b) modifying a center vanishing point gaze direction based on the x-y-z sensor inputs; and

c) Updating the non-distraction window coordinates in accordance with the modified center vanishing gaze point.

12. The method claim of 8, further comprising the step of:

modifying the maximum allowed distraction time in accordance to one or more of following factors including but not limited to total driving time since last stop, curviness of road, and weather conditions.

13. The method claim of 8, wherein a center vanishing point gaze direction is adapted to the driver, further comprising the steps of:

a) finding N face gaze directions with longest duration when the vehicle speed exceeds a certain threshold;

b) finding median of said N face directions; and

c) updating the non-distraction window coordinates in accordance with said median of said N face gaze directions of the driver, wherein camera offset angle with respect to driver'"'"'s face angle is also taken into account.

14. The method claim of 8, further comprising the steps of:

connecting to internet using a wireless connection when said at least one warning is issued; and

communicating distraction condition to a pre-determined destination which includes one or more of fleet management, parent, highway patrol, insurance for profile management, and family and friends.

15. The method claim of 8, further comprising the step of:

illuminating the driver'"'"'s face by one of methods including dim visible light and infrared light that is not visible to a human when ambient light level is low.

16. The method claim of 8, further comprising the steps of:

a) detecting the area of facial coordinates of the driver;

b) adding a padding area around the said area of facial coordinates of the driver;

c) performing auto-exposure algorithm weighted inside said padding area; and

d) updating said detected area continuously in accordance with the video stream of frames of the driver'"'"'s face.

17. The method claim of 8, wherein driver'"'"'s face gaze direction is determined using one of method including but not limited to active appearance model, cylinder-head model, appearance Template method, flexible models with active appearance models, geometric methods for facial features, tracking methods for feature tracking using affine transformation and appearance-based particle filters, and hybrid methods that includes one and more methods combined from a list of geometric method and tracking, appearance template and tracking, active appearance models, and cylinder-head models.

18. A method for a driver assistance apparatus for accident avoidance, comprising the steps of:

a) determining a speed of a vehicle;

b) performing the following steps only when said vehicle speed exceeds a predetermined speed threshold;

c) selecting a maximum allowed distracted driving distance;

d) determining said driver'"'"'s face gaze direction;

e) filtering said driver'"'"'s face gaze direction over multiple captured video frames;

f) calculating a maximum allowed distraction time in accordance with the speed of said vehicle and said selected maximum allowed distracted driving distance;

g) determining if the driver'"'"'s filtered face gaze direction is outside the non-distraction window of driver'"'"'s normal view of road ahead,

wherein taking into account of camera angle offset with respect to said driver'"'"'s face;

h) calculating a time duration during which said driver'"'"'s filtered face gaze direction is outside the non-distraction window; and

i) Issuing an alert warning to said driver when the time duration of said filtered face gaze direction exceeds said maximum allowed distraction time.

19. The method claim of 18, wherein a center vanishing point gaze direction is adapted to the driver, further comprising the steps of:

a) finding N face gaze points with longest duration when the speed of said vehicle exceeds a certain threshold value;

b) finding median of said N face points; and

c) adapting center gaze point of the driver in accordance with said median of said N gaze points.

20. The method claim of 18, further comprising the steps of:

a) connecting to internet using a wireless modem; and

b) communicating distraction condition to a pre-determined destination which includes one or more of fleet management, parent, highway patrol, insurance for profile management, and family and friends,

wherein internet protocol messaging including but not limited to short message service (SMS), email, Real Time Streaming Protocol (RTSP), hypertext transfer protocol, or file transfer protocol is used,

wherein wireless modem internet connectivity is used including but not limited to third generation (3G), fourth generation (4G) or later mobile communication technology.