Method and device for detecting drowsiness and preventing a driver of a motor vehicle from falling asleep

• US 6,304,187 B1
• Filed: 07/14/2000
• Issued: 10/16/2001
• Est. Priority Date: 01/15/1998
• Status: Expired due to Term

• Current Assignee: Image Processing Technologies, LLC (General Patent Corporation), William J. Brown
• Original Assignee: Holding B.E.V. S.A.
• Inventors: Pirim, Patrick
• Primary Examiner(s): Hofsass, Jeffery
• Assistant Examiner(s): NGUYEN, HUNG T
• Application Number:
  US09/600,394
• Time in Patent Office: 459 Days
• Field of Search: 340/575, 340/576, 340/573.1, 340/573.4, 340/691, 340/693
• US Class Current: 340/576
• CPC Class Codes: A61B 2503/22 Motor vehicles operators, e.g. drivers, pilots, captains
• CPC Class Codes: A61B 5/1103 Detecting eye twinkling
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: A61B 5/6821 Eye
• CPC Class Codes: A61B 5/6893 Cars
• CPC Class Codes: B60R 1/04 mounted inside vehicle B60R1/025, B60R1/08 take precedence
• CPC Class Codes: B60R 1/12 Mirror assemblies combined with other articles, e.g. clocks
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms

Abstract

The invention relates to a method and device using an optoelectronic sensor in combination with an electronic unit arranged inside a motor vehicle. The sensor is adjusted in line with the head of the driver sitting in said motor vehicle at the same time as the inner rear-view mirror, which comprises a one-way mirror behind which the sensor is positioned. After detecting the presence of a driver seated in the motor vehicle they frame first the whole face and then the eyes of the driver in the video signal raser emitted by the sensor aided by the electronic unit and then determine the successive durations of blinking episodes, which are compared with a limit value situated between one duration typical of an awake person and another duration typical of a drowsy person. A signal (transmitted by an alram unit) suitable for waking the driver is triggered when the duration of blinking exeeds the limit.

Claims

1. A process for continuous monitoring of the driver'"'"'s watchfulness in an automotive vehicle, in order to detect and to prevent any possible sleepiness of the former, comprising:

producing a video signal, significant in real time of the successive pictures of at least the driver'"'"'s face;

processing this signal, successively and continuously, to

detect in this signal the portion corresponding effectively to the image of the driver'"'"'s head,

determine the value of a parameter in relation to the blinking of the eyelids, with a notable modification when the driver passes from the awake condition to the sleepy condition, on either side of a threshold, and

locate in real time when the value of this parameter exceeds this significant threshold of the driver'"'"'s transition from awake condition to sleepy condition; and

triggering, in response to the threshold being exceeded, an alarm capable of waking up the driver;

and which is characterised in that

the video signal is produced by using an optoelectronic sensor interconnected with a rear-mirror on the automotive vehicle, dimensioned and arranged to essentially receive the driver'"'"'s face image when seated and whose optic axis for receiving the light rays is directed towards the driver'"'"'s head when the rear-mirror is oriented correctly; and

processing the said video signal, includes after detecting the presence of the driver on his/her seat, successively and continuously,

detecting, from an analysis of moving pixels between two successive frames of same nature of said video signal, the horizontal movements of the driver, in order to centre the face of the said driver within the successive corresponding frames of the video signal,

detecting, from an analysis of moving pixels between two successive frames of same nature of said video signal, the vertical movements in the face, thus centred, of the driver in order to centre his/her eyes,

determining, from an analysis of moving pixels between two successive frames of same nature of said video signal, the successive blinking durations of the eyes, thus centred, of the driver, whereby the said durations make up the said parameter,

comparing these successive blinking durations, thus determined, with a threshold significant of the transition from awake condition to sleepy condition of the driver, and

triggering, when the blinking durations exceed the said threshold upwards, an alarm capable of waking up the driver.

2. A process according to claim 1, characterised in that the said sensor is placed in the casing of the rear mirror behind the mirror of the former which consists of a two-way mirror, the reception optic axis (2a) of said sensor being symmetrical to an axis (2b) oriented in the vertical median plane of said vehicle, related to an orthogonal axis (6) to said two-way mirror.

3. A process according to claim 1, characterised in that the presence of the driver on his/her seat is detected while determining the number of pixels corresponding in the successive frames of the same nature in the video signal for which a significant movement is detected and by comparing this number with the total number of pixels per frame of the video signal, in order to determine whether the ratio between the number of pixels representing a displacement and the total number of pixels per frame exceeds a threshold significant of the transition from the driver'"'"'s absence on his/her seat to the presence of a driver in his/her seat.

4. A process according to claim 1, characterised in that, between the detection phases of horizontal displacements, in order to centre the driver'"'"'s face, and the detection of vertical displacements, in order to frame the driver'"'"'s eyes, a wide centring phase of the eyes has been provided while being limited to a portion of the centred face, covering the eyes and their immediate surrounding, by applying the anthropometric ratio between the said portion and the entire face of a person.

5. A process according to claim 1, characterised in that simultaneously to the determination phase of the blinking durations, a determination phase of time intervals between two successive winks has been provided and a reinforced alarm should be triggered as soon as those time intervals present an irregularity which exceeds a set threshold.

6. A process according to claim 1, characterised in that continuous update of the data is performed, regarding at least one of the following parameters:

horizontal displacements, vertical displacements, blinking durations, intervals between successive winks, in order to perfect the approximations of the normal values of these parameters for the driver actually present and in awake condition.

7. A process according to claim 1, characterised in that the different successive phases of the process are performed using successive computer programmes which would process successive values of the corresponding pixels of the frames of same nature in the video signal obtained from the said sensor.

8. A device for continuous monitoring of the driver'"'"'s watchfulness in an automotive vehicle, in order to detect and to prevent any possible sleepiness of the former, implementing the process according to claim 1 and which is characterised in that it comprises, in combination:

a) an optoelectronic sensor (10) which, in combination with associated electronic circuitry (19), prepares, in response to the reception of light rays, a video signal with frames of the same nature, or corresponding, successive frames, whereas the said sensor is interconnected with a vehicle rear mirror (8) and dimensioned and arranged to essentially receive the driver'"'"'s face image when seated and whose optic axis (10b) for receiving the light rays is directed towards the driver'"'"'s head (T) when the rear mirror is suitably oriented; and

b) at least an integrated circuit including

means to detect the driver'"'"'s presence on his/her seat in the vehicle, and to prepare a presence signal;

means, actuated by this presence signal, to detect on the basis of an analysis of moving pixels between two successive frames of same nature of said video signal, the horizontal displacements of the said driver, in order to centre his/her face (V) in the successive frames of same nature in the said video signal and to prepare a signal at the end of the face centring;

means, actuated by the said end signal of the face centring, to detect on the basis of an analysis of moving pixels between two successive frames of same nature of the portion of the successive frames of same nature in the said video signal corresponding to the face centring, the vertical displacements in the face, thus centred, of the driver, in order to centre the eyes (U) of the said driver, within the said portion of the frames of this signal and to prepare a face centring end signal of the driver'"'"'s eyes;

means, actuated by the said eyes centring end signal, to determine on the basis of an analysis of moving pixels between two successive frames of same nature of the portion of the successive frames of same nature in the said video signal corresponding to the eyes centring, the successive durations of the driver'"'"'s blinking;

means to compare these successive blinking durations, thus determined, with a threshold significant of the transition from awake condition to sleepy condition of the driver; and

means to trigger, when the blinking durations exceed the said threshold, an alarm (22) capable of waking up the driver.

9. A device according to claim 8, characterised in that the said sensor (10) is placed in the casing of the rear mirror (8) behind the mirror of the former which consists of a two-way mirror (9), said sensor being carried by a first extremity of a first rod (13) crossing, through a trunnion (17), a clamp carried by the rear mirror casing (8), inside of it, the second extremity of this rod (13) being freely articulated, with a seam (14a, 14b), to the first extremity of a second rod (12) crossing, through a trunnion (15), the rear mirror casing (8), while the second extremity of said second rod (12) is fixed to the bodywork of the vehicle (in 5) above the front wind-screen, in order that the reception optic axis (2a) of said sensor being symmetrical to an axis (2b) oriented in the vertical median plane of said vehicle, related to an orthogonal axis (6) to said two-way mirror.

10. A device according to claim 8, characterised in that the said means to detect the presence of the driver on his/her seat and to prepare a presence signal consist of means to determine the number of pixels in the successive frames of the same nature in the said video signal for which a significant movement is detected, means to compare this number with the total number of pixels per frame in the video signal, in order to determine whether the ratio between the number of pixels representing a displacement and the total number of pixels per frame exceeds a threshold significant of the transition from the driver'"'"'s absence on his/her seat to the presence of a driver in his/her seat.

11. A device according to claim 8, characterised in that it comprises moreover, means, actuated by the said face centring end signal, in order to select, in the said portion of the excessive frames of the said video signal corresponding to the face centring, a reduced portion corresponding to a wide or rough centring, of the driver'"'"'s eyes covering the eyes and their immediate surrounding by applying the anthropometric ratio between the said wide centring and the entire face of a person and means to prepare a wide centring end signal of the eyes, whereas this signal actuates the said means to detect the vertical displacements in the driver'"'"'s face.

12. A device according to claim 8, characterised in that it comprises means, operating in parallel with the said means to determine the successive blinking durations and hence actuated by the said eyes centring end signal, in order to determine the time intervals between two successive winks and to trigger a reinforced alarm as soon as these time intervals show an irregularity exceeding a set threshold.

13. A device according to claim 8, characterised in that it comprises means for continuous update of the data regarding at least one of the following parameters:

horizontal displacements, vertical displacements, blinking durations, intervals between successive winks, in order to perfect the approximations of the normal values of the parameter involved for the driver actually present and in awake condition.

14. A device according to claim 8, characterised in that the optoelectronic sensor (10)—

electronic unit (19) assembly produces a video signal comprising a succession of corresponding frames of same nature with a succession of lines made of a succession of pixels and processes the said video signal in order to successively

deduct, from the variations in the value or intensity of each pixel between a frame and the corresponding former frame

on the one hand, a binary signal, noted DP, whose both possible values are significant, one of a variation in the pixel value and the other, of a non-variation of the pixel value and

on the other, a digital signal, noted CO, with a reduced number of possible values, whereby this signal is significant of the magnitude of this variation in the pixel value;

distribute according to a matrix, on a noria basis, values for both these signals DP and CO for the same frame scrolling through the matrix; and

derive, from this matrix distribution, the requested displacement and its location and direction parameters.

15. A device according to claim 8, characterised in that said sensor (10), said associated electronic circuitry (19) and said integrated circuit are constituted by an electronic chip arranged in the rear mirror casing (8).

16. A rear mirror for automotive vehicle, characterised in that its mirror consists of a two-way mirror (9) and in that it comprises, behind this two-way mirror, a sensor fitted with optoelectronic circuitry (10) working together with an electronic unit (19) also provided inside the rear mirror and in that the optoelectronic sensor (10)—

electronic unit (19) assembly produces a video signal comprising a succession of corresponding frames of same nature with a succession of lines made of a succession of pixels and processes the said video signal in order to successively

deduct, from the variations in the value or intensity of each pixel between a frame and the corresponding former frame

a binary signal, noted DP, whose both possible values are significant, one of a variation in the pixel value and the other, of a non-variation of the pixel value, and

a digital signal, noted CO, with a reduced number of possible values, whereby this signal is significant of the magnitude of this variation in the pixel value;

distribute according to a matrix, on a noria basis, values for both these signals DP and CO for the same frame scrolling through the matrix; and

derive, from this matrix distribution, the requested displacement and its location and direction parameters; and

trigger an alarm device (22) as soon as the said unit determines that the vertical movements of a person'"'"'s eyelids looking at the front face (9a) of the said mirror correspond to a blinking duration exceeding a pre-set threshold included in the time interval comprised between the duration a woken person'"'"'s blinking and that of a person falling asleep.

17. A rear mirror for automotive vehicle according to claim 16, characterised in that said sensor (10), said associated electronic circuitry (19) and said integrated circuit are constituted by an electronic chip arranged in the rear mirror casing (8).

18. A rear mirror for automotive vehicle according to claim 16, characterised in that it carries moreover at least one light-emitting diode (20) at least in the infrared range which is actuated at least when the surrounding luminosity becomes insufficient to light the driver'"'"'s face and in that the said optoelectronic sensor (10) is sensitive, among other things, to the infrared radiations transmitted by the said diode.