Device, program, and method for determining sleepiness

• US 20080212828A1
• Filed: 02/12/2008
• Published: 09/04/2008
• Est. Priority Date: 02/16/2007
• Status: Active Grant

• Current Assignee: DENSO Corporation
• Original Assignee: DENSO Corporation
• Inventors: Ishida, Kenji, Ozaki, Noriyuki, Uchiyama, Kahori, Kimura, Teiyuu
• Granted Patent: US 8,045,766 B2
• Time in Patent Office: Days
• US Class Current: 382/100
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A sleep prevention system captures a facial image of a driver, determines a sleepiness level from the facial image and operates warning devices including neck air conditioner, seatbelt vibrator, and brake controller if necessary based on the sleepiness determination. A sleepiness determination device determines sleepiness from facial expression information such as distances between corners of a mouth, distance between an eyebrow and eye, tilt angle of a head, and other facial feature distances. The facial distances are calculated from captured images and from reference information gather during wakefulness. The sleepiness degree is determined based on the determination results including combinations thereof.

Claims

1. A sleepiness determination device comprising:

facial expression information obtaining means for obtaining at least two of facial expression information elements (A) to (K) associated with a facial image including;

(A) a distance between left and right corners of a mouth;

(B) a distance between an eyebrow and eye;

(C) a tilt angle of a head;

(D) a distance between the midpoint between inner corners of eyebrows and the midpoint between inner corners of eyes;

(E) a distance between inner corners of eyebrows;

(F) a distance between the midpoint between outer corners of a lower nose and the midpoint between inner corners of eyes;

(G) a distance between the midpoint between corners of a mouth and the midpoint between inner corners of eyes;

(H) a distance between a lower edge of a lower lip and the midpoint between inner corners of eyes;

(I) a distance between upper and lower eyelids;

(J) a distance between outer corners of a lower nose; and

(K) a distance between an upper edge of an upper lip and the midpoint between inner corners of eyes; and

sleepiness determination means for;

determining a respective indication of sleepiness for each of the at least two facial expression information elements; and

determining a level of sleepiness based on the respective indication of sleepiness for the each.

2. The sleepiness determination device of claim 1, wherein the sleepiness determination means determines the respective indication of sleepiness by comparing the facial expression information obtained during wakefulness and the at least two of the facial expression information elements (A) to (K) obtained at a time at which sleepiness is to be determined, and determines the level of sleepiness from a combination of results of the determination.

3. The sleepiness determination device of claim 1, wherein the facial expression information obtaining means further detects positional and angular information associated with the facial image, and obtains the at least two of the facial expression information elements (A) to (K) in accordance with the positional and angular information.

4. The sleepiness determination device of claim 3, wherein the sleepiness determination device is mounted and used in a vehicle, and the facial expression information obtaining means detects the positional and angular information based on the facial image data captured by imaging means provided to the vehicle.

5. The sleepiness determination device of claim 1, wherein the sleepiness determination means determines one of a presence and an absence of sleepiness based on the at least two of the facial expression information elements (A) to (K) obtained by the facial expression information obtaining means.

6. The sleepiness determination device of claim 1, wherein the sleepiness determination means determines the level of sleepiness based on a ratio of the at least two of the facial expression information elements (A) to (K) showing the presence of sleepiness relative to reference facial expression information obtained during wakefulness.

7. The sleepiness determination device of claim 1, wherein:

the sleepiness determination means obtains the (C) tilt angle of the head in addition to the at least two least two of the facial expression information elements (A) to (K) other than the (C) tilt angle of the head, the at least two least two of the facial expression information elements (A) to (K) including the (A) distance between the left and right corners of the mouth and the (B) distance between the eyebrow and the eye;

a first condition includes that sleepiness is determined to be present based on the (A) distance between left and right corners of a mouth, a second condition includes that sleepiness is determined to be present based on the distance (B) between an eyebrow and eye and based on the first condition, and a third condition includes that sleepiness is determined to be present based on the (C) tilt angle of a head and based on the first and second conditions, and

the level of sleepiness is determined as slight sleepiness when the only first condition is satisfied, the level of sleepiness is determined as moderate sleepiness higher than the slight sleepiness when only the second condition is satisfied and the level of sleepiness is determined as strong sleepiness higher than the moderate sleepiness when only the third condition is satisfied, and the level of sleepiness is determined as no sleepiness when none of the conditions are satisfied.

8. An article of manufacture comprising program instructions carried on a computer readable medium, the instructions when read for causing a processor in a computer system to execute a procedure to carry out the functions of the facial expression information obtaining means and sleepiness determination means in accordance with claim 1.

9. A sleepiness determination method comprising’

obtaining at least two of a distance between left and right corners of a mouth, a distance between an eyebrow and eye, a tilt angle of a head;

a distance between a midpoint between inner corners of eyebrows and a midpoint between inner corners of eyes, a distance between inner corners of eyebrows, a distance between a midpoint between corners of a lower nose and a midpoint between inner corners of eyes, a distance between a midpoint between outer corners of a mouth and a midpoint between inner corners of eyes, a distance between a lower edge of a lower lip and a midpoint between inner corners of eyes, a distance between upper and lower eyelids, a distance between left and right corners of a lower nose, and a distance between an upper edge of an upper lip and the midpoint between inner corners of eyes; and

determining a presence or absence of sleepiness from the two or more pieces of the information or levels of sleepiness, and for determining a degree of sleepiness from combination of results of the determination.

10. A sleepiness determination device comprising:

a processor; and

a memory coupled to the processor, the memory storing facial expression information obtained from a reference facial image obtained during a wakefulness state;

facial expression information obtaining means for obtaining at least two facial expression information elements associated with a facial image obtained during operation of a vehicle; and

sleepiness determination means for;

determining a respective indication of sleepiness for each of the at least two facial expression information elements by comparing the each of the at least two facial expression information elements during operation of the vehicle with the stored facial expression information; and

determining a level of sleepiness based on a combination of the respective indication for the each.

11. The sleepiness determination device of claim 10, wherein the facial expression information elements include:

(A) a distance between left and right corners of a mouth;

(B) a distance between an eyebrow and eye;

(C) a tilt angle of a head;

(D) a distance between the midpoint between inner corners of eyebrows and the midpoint between inner corners of eyes;

(E) a distance between inner corners of eyebrows;

(F) a distance between the midpoint between outer corners of a lower nose and the midpoint between inner corners of eyes;

(G) a distance between the midpoint between corners of a mouth and the midpoint between inner corners of eyes;

(H) a distance between a lower edge of a lower lip and the midpoint between inner corners of eyes;

(I) a distance between upper and lower eyelids;

(J) a distance between outer corners of a lower nose; and

(K) a distance between an upper edge of an upper lip and the midpoint.

12. The sleepiness determination device of claim 10, wherein:

the respective indication of sleepiness associated with a first condition is determined based on the (A) distance between left and right corners of a mouth, the respective indication of sleepiness associated with a second condition is determined based on the distance (B) between an eyebrow and eye and based on the first condition, and the respective indication of sleepiness associated with a third condition is determined based on the (C) tilt angle of a head and based on the first and second conditions, and

the level of sleepiness is determined as slight sleepiness when the only first condition is satisfied, the level of sleepiness is determined as moderate sleepiness higher than the slight sleepiness when only the second condition is satisfied and the level of sleepiness is determined as strong sleepiness higher than the moderate sleepiness when only the third condition is satisfied, and the level of sleepiness is determined as no sleepiness when none of the conditions are satisfied.