TWO-STEP SLEEPY DRIVING PREVENTION APPARATUS THROUGH RECOGNIZING OPERATION, FRONT FACE, EYE, AND MOUTH SHAPE

• US 20160272217A1
• Filed: 10/27/2014
• Published: 09/22/2016
• Est. Priority Date: 10/29/2013
• Status: Abandoned Application

• Current Assignee: Jae-Chul Kim
• Original Assignee: Jae-Chul Kim
• Inventors: KIM, Jae-Chul, KO, Jae-Kwan, KIM, Woo-Chang, LEE, Chang-Hui, PARK, Su-Jin
• Application Number:
  US15/032,695
• Publication Number:
  US 20160272217A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B60K 28/06 responsive to incapacity of driver
• CPC Class Codes: B60K 28/066 actuating a signalling device B60K28/063 takes precedence
• CPC Class Codes: B60Q 9/00 Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 , e.g. haptic signalling
• CPC Class Codes: B60W 2040/0827 due to sleepiness
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2540/22 Psychological state; Stress level or workload
• CPC Class Codes: B60W 2540/221 Physiology, e.g. weight, heartbeat, health or special needs
• CPC Class Codes: B60W 2556/50 of positioning data, e.g. GPS [Global Positioning System] data
• CPC Class Codes: B60W 40/09 Driving style or behaviour
• CPC Class Codes: G01S 19/13 Receivers
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/169 Holistic features and representations, i.e. based on the facial image taken as a whole
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

Disclosed therein is a two-step drowsy driving prevention apparatus through motion, face, eye, and mouth recognition. A driving judging unit which includes a GPS device to detect a driving state of a vehicle and a drowsiness judging unit which receives image and three-dimensional depth information from a photographing unit including an infrared depth sensor and an infrared camera mounted at positions to observe a driver'"'"'s face judge a cycle of a yawn, a speed and a cycle of nodding the head and a speed and a cycle of a blink of the eye as a first step judgment. When it is judged that there is a drowsiness prelude action, the apparatus outputs a “preliminary warning”. After that, a warning unit outputs a “main warning” when there is a drowsiness execution action by inspecting a closed state of the eye, a forward looking state of the eye and an angle of the head as a second step judgment so as to accurately judge only a drowsy driving action except the closed state of the eye and the case that the driver does not look at the front or bends the head other than drowsiness through the two-step stepwise judging method based on various judgment criteria, such as the driver'"'"'s motion, face, pupil and a mouth shape, thereby preventing traffic accidents, human casualties and economic loss due to drowsy driving.

Claims

1. A two-step drowsy driving prevention apparatus through motion, face, pupil and mouth recognition, the apparatus comprising:

a driving judging unit to judge a forward driving state of a vehicle;

a photographing unit to photograph a driver'"'"'s condition;

a drowsiness judging unit to judge whether or not the driver is driving while drowsiness;

a warning unit to generate a warning sound; and

a controller to control the driving judging unit, the photographing unit, the drowsiness judging unit and the warning unit,

wherein the driving judging unit comprises;

a first-step judging process to judge a drowsiness prelude action through recognition of the driver'"'"'s motion, face, pupil and mouth shape only when the vehicle is moving forward by receiving coordinate information of the current location of the vehicle from a GPS terminal and comparing the coordinate information with coordinate information of location for several seconds before the current location in order to judge whether the vehicle is driving forward, is driving backward or is stopped; and

a second step judging process to judge a drowsiness execution action, so that the drowsy driving prevention apparatus finally determines that the driver is in drowsy driving and generates a warning sound only when it is judged that there is a drowsiness execution action in the second step determination process after it is determined that there is the drowsiness prelude action of the first step.

2. The two-step drowsy driving prevention apparatus according to claim 1, wherein the photographing unit to photograph the driver'"'"'s condition comprises an infrared camera to receive image information of the driver'"'"'s facial state and an infrared depth sensor to receive three-dimensional depth information

3. The two-step drowsy driving prevention apparatus according to claim 2, wherein the infrared camera receives the image information of the facial state and transmits the image information of the facial state to the drowsiness judging unit to recognize the driver'"'"'s face outline, eye outline, white and iris of the eye in a bright area as well as in a dark area.

4. The two-step drowsy driving prevention apparatus according to claim 2, wherein the infrared depth sensor receives the three-dimensional depth information and transmits the three-dimensional depth information to the drowsiness judging unit to accurately judge locations and movements of the driver'"'"'s upper body, head, face, eye and a mouth in a bright area as well as in a dark area.

5. The two-step drowsy driving prevention apparatus according to claim 1, wherein the drowsiness judging unit to judge the driver'"'"'s drowsiness state is mounted to judge the drowsy driving state by receiving the image information from the infrared camera and the three-dimensional depth information from the infrared depth sensor only when the driving judging unit judges a forward driving state, and first recognizes whether or not the driver carries out any prelude action of drowsiness through the first-step judging process and judges whether or not the driver carries out drowsiness execution action through the second-step judging process, and then, finally determines drowsy driving only when going through the second step after passing the first step.

6. The two-step drowsy driving prevention apparatus according to claim 5, wherein the first step to judge the prelude action of the drowsiness judges the prelude action of the drowsiness by calculating a cycle of driver'"'"'s yawn or calculating a speed and a cycle of nodding the head after first detecting the driver'"'"'s face outline from image and sensor information transmitted from the photographing unit and detecting locations of the mouth, the head, the nose and the neck, or by calculating a speed and a cycle of blinking eyes after first detecting locations of the eyes and recognizing the outlines of the pupils and the whites and the irises of the pupils.

7. The two-step drowsy driving prevention apparatus according to claim 6, wherein the cycle of yawning, which is one of the criteria to judge the prelude action of the drowsiness uses the image and the sensor information received from the photographing unit to identify the location of the driver'"'"'s mouth and to grasp whether or not the driver yawns based on the shape of the mouth or the degree of opening of the lower jaw, thereby judging that there is a drowsiness prelude action when the number of yawning cycles exceeds the standard of the yawning cycles within a predetermined period of time.

8. The two-step drowsy driving prevention apparatus according to claim 6, wherein the speed and the cycle of nodding the head, which is one of the criteria of judging the prelude action of the drowsiness uses the image and the sensor information received from the photographing unit to regard as one nod when the driver'"'"'s head bends more than a predetermined angle and returns to its initial state, so that the drowsy driving prevention apparatus calculates as a “

nod as the prelude of drowsiness”

only when one nod of the head spends more time than the standard time and judges that the prelude action of the drowsiness exists when the nod as the prelude of the drowsiness occurs more than a reference number of nods within a predetermined time.

9. The two-step drowsy driving prevention apparatus according to claim 6, wherein the period of the yawn, which is one of a criteria of judging the prelude action of the drowsiness, uses the image and the sensor information received from the photographing unit to regard that the driver'"'"'s pupils are covered more than a predetermined portion and then uncovered as one blink, so that the drowsy driving prevention apparatus calculates as a “

blink as the prelude of drowsiness”

only when one blink of the eyes spends more time than the standard time and judges that the prelude action of the drowsiness exists when the blink as the prelude of the drowsiness occurs more than a reference number of blinks within a predetermined time.

10. The two-step drowsy driving prevention apparatus according to claim 5, wherein the second step to judge the execution action state of the drowsiness judges that there is an execution action of the drowsiness when the driver who satisfies one of the criteria of the drowsiness prelude action of the first step closes the eyes for more than a predetermined period of time when the driver does not look at the front for more than a predetermined period of time or when the driver bends his or her head at the predetermined angle for more than a predetermined period of time.

11. The two-step drowsy driving prevention apparatus according to claim 10, wherein the closed state of the driver'"'"'s eye, which is one of the criteria to judge the drowsiness execution action uses the image information received from the photographing unit to judge if the driver'"'"'s pupil is closed more than a predetermined degree while sustaining the closed state for more than a predetermined time by analyzing the driver'"'"'s pupil.

12. The two-step drowsy driving prevention apparatus according to claim 10, wherein the forward looking of the driver'"'"'s eye, which is one of the criteria to judge the drowsiness execution action uses the image information received from the photographing unit to judge the drowsiness execution step when the time that the driver does not look at the front is continued for more than a predetermined period of time by analyzing the driver'"'"'s pupil.

13. The two-step drowsy driving prevention apparatus according to claim 10, wherein the angle of the head, which is one of the criteria to judge the drowsiness execution action uses the image information received from the photographing unit to judge as the drowsiness execution step when the head is bent at more than a predetermined angle and continues the bent state for more than a reference time by calculating an angle between a reference point of the driver'"'"'s head and a reference point of the driver'"'"'s nose.

14. The two-step drowsy driving prevention apparatus according to claim 1, wherein the warning unit, which warns the user, informs the driver of the drowsy driving state by generating a preliminary warning sound when the drowsiness prelude action of the first step is judged and generating the main warning sound when the drowsiness execution action of the second step is judged after passing the first step.

15. The two-step drowsy driving prevention apparatus according to claim 1, wherein the controller to control the driving judging unit, the photographing unit, the drowsiness judging unit and the warning unit receives the forward driving state from the driving judging unit, transmits a photograph stop command to the photographing unit when the vehicle is stopped or driving backward, and transmits a first step drowsiness judging command to the drowsiness judging unit only when the vehicle is driving forward, sends a detection command to the photograph unit, transmits the second step drowsiness judging command to the drowsiness judging unit, and instructs the warning unit to send the preliminary warning sound after receiving information that the first step drowsiness prelude action exists from the drowsiness judging unit and instructs the warning unit to send the main warning sound to the driver when receiving information that the second step execution action of the drowsiness exists after passing the first step.