Eye closure detection using structured illumination

• US 8,314,707 B2
• Filed: 03/30/2010
• Issued: 11/20/2012
• Est. Priority Date: 03/30/2009
• Status: Active Grant

• Current Assignee: Tobii Technology AB
• Original Assignee: Tobii Technology AB
• Inventors: Kobetski, Miroslav, Blixt, Peter, Holmberg, Johnny, Elvesjo, John Mikael Holtz
• Primary Examiner(s): Lee, Benjamin C
• Assistant Examiner(s): Carlson, Adam
• Application Number:
  US12/749,612
• Publication Number:
  US 20100245093A1
• Time in Patent Office: 966 Days
• Field of Search: 340/576, 340/575, 348/78, 382100-103
• US Class Current: 340/575
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: B60K 28/066 actuating a signalling device B60K28/063 takes precedence
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: Y02T 10/84 Data processing systems or methods, management, administration

Abstract

A monitoring system monitors and/or predicts drowsiness of a driver of a vehicle or a machine operator. A set of infrared or near infrared light sources is arranged such that an amount of the light emitted from the light source strikes an eye of the driver or operator. The light that impinges on the eye of the driver or operator forms a virtual image of the signal sources on the eye, including the sclera and/or cornea. An image sensor obtains consecutive images capturing the reflected light. Each image contains glints from at least a subset or from all of the light sources. A drowsiness index can be determined based on the extracted information of the glints of the sequence of images. The drowsiness index indicates a degree of drowsiness of the driver or operator.

Claims

1. A monitoring system for monitoring and/or predicting drowsiness of a driver of a vehicle or a machinery operator, the system being adapted to be arranged in the vehicle or at the machinery, comprising:

a set of infrared or near infrared light sources, each light source being arranged in the vehicle or at the machinery such that a substantial amount of the light emitted from the light source incidents on an eye of the driver or operator, the emitted light thereby forming a virtual image of the signal source on the eye including the cornea;

an image sensor arranged and oriented such that light reflected from the eye can be captured in an image and adapted to generate consecutive images forming a sequence of images over time, wherein each image contains glints from at least a subset of the light sources;

an image processing unit adapted to extract information of the glints in the images by processing the images using an image processing algorithm, said image processing unit being adapted to identify glints in the image corresponding to reflected light from respective light sources, with the extracted information including a measure of glints being proportional to a number of glints identified in an image; and

a determining unit adapted to determine a drowsiness index based on the extracted information of the glints of the sequence of images, wherein the drowsiness index indicates a degree of drowsiness.

2. The system according to claim 1, wherein said determining unit is adapted to:

determine an amplitude measure for each image or a number of images based on the identified glints, wherein the amplitude measure is based on at least one of a frequency measure, a spatial measure, and a wavelet based measure; and

determine the drowsiness index using the amplitude measure.

3. The system according to claim 2, wherein said determining unit is adapted to calculate at least one of the following parameters a)-h):

a) a value of the amplitude measure during a predetermined period of time corresponding to a predetermined number of images;

b) negative time derivative of the amplitude measure during the predetermined period of time;

c) a time derivative during the predetermined period of time;

d) a period of time between two consecutive amplitude values of the amplitude measure;

e) a period of time when the amplitude measure is below a predetermined threshold during the predetermined period of time;

f) a period of time when the amplitude measure is above a predetermined threshold during the predetermined period of time;

g) a time derivative profile of the amplitude measure of a sub-period of the predetermined period of time; and

h) a morphology of the amplitude measure during the predetermined period of time; and

wherein said determining unit is adapted to base the drowsiness index on at least one of the parameters a)-h).

4. The system according to claim 3, wherein said determining unit is adapted to determine the drowsiness index, based on at least one of the parameters a)-h), to correspond to:

a relation between a blinking amplitude and a maximum closure speed;

a blinking frequency; and

a period of time during which the eye is closed.

5. The system according to claim 4, further including an alert system, and wherein said determining unit is adapted to instruct said alert system to notify the driver or operator of an increased degree of drowsiness when the drowsiness index has exceeded the predetermined drowsiness index threshold, the notification being delivered by an alert signal comprising at least one of an alert tone, an alert sound, an alert light signal and a vibration signal.

6. The system according to claim 1, wherein said determining unit is adapted to alert the driver or operator of an increased degree of drowsiness when the drowsiness index exceeds a predetermined drowsiness index threshold.

7. The system according to claim 1, wherein the infrared or near infrared light is modulated light;

and said image processing unit is adapted to discriminate reflections of the infrared or near infrared light from other light reflections captured in the images based on the modulation.

8. The system according to claim 7, wherein the modulated infrared or near infrared light is modulated with respect to at least one quantity in a group comprising time, wavelength and polarization.

9. The system according to claim 1, wherein the set of infrared (IR) or near infrared (NIR) light sources are configured to emit infrared or near infrared light at a specific wavelength.

10. A method for monitoring and/or predicting drowsiness of a driver of a vehicle or a machinery operator, comprising:

emitting light using a set of infrared or near infrared light sources, each light source being arranged in the vehicle or at the machinery such that a substantial amount of the light emitted from said light source incidents on an eye of the driver or operator, the emitted light thereby forming a virtual image of said signal source on the eye including the cornea;

capturing light reflected from the eye in an image using an image sensor;

generate consecutive images forming a sequence of images over time, wherein each image contains glints from at least a subset of said light sources;

processing the images to extract information of the glints in the image corresponding to reflected light from respective light sources, with the extracted information including a measure of glints being proportional to a number of glints identified in an image; and

determining a drowsiness index based on the extracted information of the glints of the sequence of images, wherein the drowsiness index indicates a degree of drowsiness.

11. The method according to claim 10, further comprising:

determining an amplitude measure for each image or a number of images based on the identified glints, wherein the amplitude measure is based on a frequency measure, a spatial measure or a wavelet based measure; and

determining the drowsiness index using the amplitude measure.

12. The method according to claim 11, further comprising calculating at least one of the following parameters a)-h):

a) a maximum value of the amplitude measure during a predetermined period of time corresponding to a predetermined number of images;

b) a maximum negative time derivative of the amplitude measure during the predetermined period of time;

c) a maximum time derivative during the predetermined period of time;

d) a period of time between two consecutive maximum amplitude values of the amplitude measure;

e) a period of time when the amplitude measure is below a predetermined threshold during the predetermined period of time;

f) a period of time when the amplitude measure is above a predetermined threshold during the predetermined period of time;

g) a time derivative profile of the amplitude measure of a sub-period of the predetermined period of time;

h) a morphology of the amplitude measure during the predetermined period of time; and

wherein determining the drowsiness index is based on at least one of the parameters a)-h).

13. The method according to claim 12, further comprising determining the drowsiness index, based on at least one of the parameters a)-h), to correspond to a relation between a blinking amplitude and a maximum closure speed, a blinking frequency or a period of time during which the eye is closed.

14. The method according to claim 10, further comprising alerting the driver or operator of an increased degree of drowsiness when the drowsiness index exceeds a predetermined drowsiness index threshold.

15. The method according to claim 14, further including notifying the driver or operator of an increased degree of drowsiness when the drowsiness index has exceeded the predetermined drowsiness index threshold, the notification being delivered by an alert signal including at least one of an alert tone, an alert sound, an alert light signal, and a vibration signal.

16. The method according to any claim 10, further comprising:

modulating the light emitted from said light sources; and

discriminating reflections of said infrared or near infrared light from other light reflections captured in the images based on the modulation.

17. The method according to claim 16, wherein the modulated infrared or near infrared light is modulated with respect to at least one quantity in a group consisting of time, wavelength and polarization.

18. The method according to claim 10, further comprising emitting light using the set of infrared (IR) or near infrared (NIR) light sources at a specific wavelength.