Impaired operator detection and warning system employing eyeblink analysis

• US 5,867,587 A
• Filed: 05/19/1997
• Issued: 02/02/1999
• Est. Priority Date: 05/19/1997
• Status: Expired due to Term

• Current Assignee: MedFlex LLC
• Original Assignee: Northrop Grumman Corporation
• Inventors: Aboutalib, Omar, Ramroth, Richard Roy
• Primary Examiner(s): Au, Amelia
• Assistant Examiner(s): Bali, Vikkram
• Application Number:
  US08/858,771
• Time in Patent Office: 624 Days
• Field of Search: 382/115, 382/278, 382/116, 382/117, 382/291, 364/819, 364/820, 340/576, 340/573.7, 340/575, 340/439, 351/209, 351/210, 351/246
• US Class Current: 382/117
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms

Abstract

A system and method for detecting and warning of an impaired operator, such as a drowsy vehicle/machine operator or air traffic controller. The system and method employ an imaging apparatus which produces consecutive digital images including the face and eyes of an operator. There is an eye finding unit which determines the location of the operator'"'"'s eyes within each digital image, and generates correlation coefficients corresponding to each eye which quantify the degree of correspondence between pixels associated with the location of the operator'"'"'s eye in an immediately preceding image in comparison to pixels associated with the location of the operator'"'"'s eye in a current image. An impaired operator detection unit is used to average the first N consecutive correlation coefficients generated to generate a first average correlation coefficient. After the production of each subsequent image by the imaging apparatus, the impaired operator detection unit averages the previous N consecutive correlation coefficients generated to create a next average correlation coefficient. Next, the impaired operator detection unit analyzes the average correlation coefficients associated with each eye to extract at least one parameter attributable to an eyeblink of the operator'"'"'s eyes. These extracted parameters are compared to an alert operator threshold associated with that parameter. An impaired operator warning unit is used to indicate that the operator may be impaired if any extracted parameters deviate from the associated threshold in a prescribed way.

Claims

1. A method of detecting an impaired operator, comprising the steps of:

(a) employing an imaging apparatus which produces consecutive digital images including the face and eyes of an operator, each digital image comprising an array of pixel representing the intensity of light reflected from the face of the subject;

(b) determining the location of a first one of the operator'"'"'s eyes within each digital image;

(c) generating correlation coefficients, each of which quantifies the degree of correspondence between pixels associated with the location of the operator'"'"'s eye in an immediately preceding image in comparison to pixels associated with the location of the operator'"'"'s eye in a current image;

(d) averaging the first N consecutive correlation coefficients generated to generate a first average correlation coefficient, wherein N corresponds to at least the number of images required to image a blink of the operator'"'"'s eye;

(e) after the production of the next image by the imaging apparatus, averaging the previous N consecutive correlation coefficients generated to create a next average correlation coefficient;

(f) repeating step (e) for each image frame produced by the imaging apparatus;

(g) analyzing said average correlation coefficients to extract at least one parameter attributable to an eyeblink of said operator'"'"'s eye;

(h) comparing each extracted parameter to an alert operator threshold associated with that parameter, said threshold being indicative of an alert operator; and

(i) indicating that the operator may be impaired if any extracted parameter deviates from the associated threshold in a prescribed way.

2. The method of claim 1, further comprising the steps of:

(j) determining the location of the other of the operator'"'"'s eyes within each digital image; and

(k) performing steps (c) through (i) for the location of the other of the operator'"'"'eyes.

3. The method of claim 1, wherein:

the analyzing step comprises extracting a parameter indicative of the duration of an operator'"'"'s eyeblinks;

the comparing step comprises comparing the extracted duration parameter to said associated alert operator threshold wherein the threshold corresponds to a maximum eyeblink duration expected to be exhibited by an alert operator'"'"'s eye; and

the indicating step comprises indicating the operator may be impaired whenever the extracted duration parameter exceeds the alert operator duration threshold.

4. The method of claim 3, wherein:

the step of extracting the duration parameter comprises identifying each average correlation coefficient generated;

the step of comparing the extracted duration parameter to said associated alert operator duration threshold comprises comparing each average correlation coefficient to a minimum average correlation coefficient value which is would be obtained by averaging the correlation coefficients generated for N images capturing a complete blink of an alert operator'"'"'s eye; and

wherein,

the extracted duration parameter exceeds the alert operator threshold whenever the average correlation coefficients are less than the minimum average correlation coefficient value which is would be obtained by averaging the correlation coefficients generated for N images capturing a complete blink of an alert operator'"'"'s eye.

5. The method of claim 1, wherein:

the analyzing step comprises extracting a parameter indicative of the frequency of an operator'"'"'s eyeblinks;

the comparing step comprises comparing the extracted frequency parameter to said associated alert operator threshold wherein the threshold corresponds to a minimum eyeblink frequency expected to be exhibited by an alert operator'"'"'s eye; and

the indicating step comprises indicating the operator may be impaired whenever the extracted frequency parameter is less than the alert operator frequency threshold.

6. The method of claim 1, wherein the step of extracting the frequency parameter comprises the steps of counting the number of minimum average correlation coefficients occurring over a prescribed preceding period of time which are less than a prescribed value, and thereafter dividing the counted number of minimum average correlation coefficients by the prescribed period of time to determine an eyeblink frequency, wherein said prescribed value corresponds a maximum average correlation coefficient value which is would be obtained by averaging the correlation coefficients generated for N images capturing a complete blink of an alert operator'"'"'s eye, said analyzing step being repeated each time an average correlation coefficient is generated.

7. The method of claim 1, wherein:

the analyzing step comprises extracting a parameter indicative of the amplitude of an operator'"'"'s eyeblinks;

the comparing step comprises comparing the extracted amplitude parameter to said associated alert operator threshold wherein the threshold corresponds to a minimum eyeblink amplitude expected to be exhibited by an alert operator'"'"'s eye; and

the indicating step comprises indicating the operator may be impaired whenever the extracted amplitude parameter is less than the alert operator amplitude threshold.

8. The method of claim 1, wherein the step of extracting the amplitude parameter comprises the steps of identifying each occurrence of a minimum average correlation coefficient which is less than a prescribed value and for each such occurrence determining the absolute value of the difference between the minimum average correlation coefficient and the next occurring maximum average correlation coefficient, wherein said absolute value is indicative of the amplitude of an operator'"'"'s eyeblink, and wherein said prescribed value corresponds a maximum average correlation coefficient value which is would be obtained by averaging the correlation coefficients generated for N images capturing a complete blink of an alert operator'"'"'s eye.

9. The method of claim 2, wherein

the comparing step comprises first determining the difference between an extracted parameter associated with the first eye and a like extracted parameter associated with the other eye to establish a parameter consistency factor for the extracted parameter, and thereafter comparing the established parameter consistency factor to an alert operator consistency threshold associated with that parameter.

10. The method of claim 9, wherein:

the analyzing step comprises extracting a parameter indicative of the duration of an operator'"'"'s eyeblinks;

the comparing step comprises determining the difference between the duration of each of the operator'"'"'s eyeblinks associated with the first eye to the duration of a next occurring eyeblink associated with the other eye to establish a duration consistency factor, and thereafter comparing the determined duration consistency factor to said associated alert operator consistency threshold wherein the threshold corresponds to a minimum difference in the eyeblink duration expected to be exhibited by an alert operator'"'"'s eyes; and

the indicating step comprises indicating that the operator may be impaired whenever the determined duration consistency factor exceeds the associated alert operator duration consistency threshold.

11. The method of claim 9, wherein:

the analyzing step comprises extracting parameters indicative of the frequency of an operator'"'"'s eyeblinks;

the comparing step comprises determining the difference between the frequency of the operator'"'"'s eyeblinks associated with the first eye as calculated over a prescribed period of time to the contemporaneous frequency of the eyeblinks associated with the other eye as calculated for the prescribed period of time to establish a frequency consistency factor, and thereafter comparing the determined frequency consistency factor to said associated alert operator consistency threshold wherein the threshold corresponds to a minimum difference in the eyeblink frequency expected to be exhibited by an alert operator'"'"'s eyes; and

the indicating step comprises indicating that the operator may be impaired whenever the determined frequency consistency factor exceeds the alert operator frequency consistency threshold.

12. The method of claim 9, wherein:

the analyzing step comprises extracting parameters indicative of the amplitude of an operator'"'"'s eyeblinks;

the comparing step comprises determining the difference between the amplitude of each of the operator'"'"'s eyeblinks associated with the first eye to the completeness of the next occurring eyeblink associated with the other eye to establish a amplitude consistency factor, and thereafter comparing the determined amplitude consistency factor to said associated alert operator consistency threshold wherein the threshold corresponds to a minimum difference in the amplitude of eyeblinks expected to be exhibited by an alert operator'"'"'s eyes; and

the indicating step comprises indicating that the operator may be impaired whenever the determined amplitude consistency factor exceeds the alert operator amplitude consistency threshold.

13. The method of claim 2, wherein plural parameters attributable to an eyeblink of said operator'"'"'s eye are extracted, and wherein:

the analyzing step comprises extracting parameters indicative of the duration, frequency, and amplitude of an operator'"'"'s eyeblinks;

the comparing step further comprises;

comparing the extracted duration parameter to an alert operator duration threshold wherein the duration threshold corresponds to a maximum eyeblink duration expected to be exhibited by an alert operator'"'"'s eye,

comparing the extracted frequency parameter to an alert operator frequency threshold wherein the frequency threshold corresponds to a minimum eyeblink frequency expected to be exhibited by an alert operator'"'"'s eye,

comparing the extracted amplitude parameter to an alert operator amplitude threshold wherein the amplitude corresponds to a minimum eyeblink amplitude expected to be exhibited by an alert operator'"'"'s eye, and

determining the difference between at least one of the extracted parameters associated with the first eye and a like extracted parameter associated with the other eye to establish a consistency factor for the extracted parameter, and thereafter comparing the established parameter consistency factor to an alert operator consistency threshold associated with that parameter.

14. The method of claim 13, wherein the indicating step comprises indicating the operator may be impaired whenever at least one of (1) the extracted duration parameter exceeds the alert operator duration threshold, (2) the extracted frequency parameter is less than the alert operator frequency threshold, (3) the extracted amplitude parameter is less than the alert operator amplitude threshold, and (4) an established parameter consistency factor exceeds an associated alert operator consistency threshold.

15. The method of claim 1, further comprising the step of:

(j) generating a corroborating indicator of operator impairedness whenever operator control inputs are indicative of the operator being impaired; and

(k) indicating that the operator is impaired if at least one of the extracted parameter deviates from the associated threshold in a prescribed way, and the corroborating indicator is generated.

16. An impaired operator detection and warning system, comprising:

an imaging apparatus which produces consecutive digital images including the face and eyes of an operator, each digital image comprising an array of pixel representing the intensity of light reflected from the face of the subject;

an eye finding unit comprising an eye finding processor, said eye finding processor comprising;

a first processor portion capable of determining the location of a first one of the operator'"'"'s eyes within each digital image,

a second processor portion capable of generating correlation coefficients each of which quantifies the degree of correspondence between pixels associated with the location of the operator'"'"'s eye in an immediately preceding image in comparison to pixels associated with the location of the operator'"'"'s eye in a current image;

an impaired operator detection unit comprising an impaired operator detection processor, said impaired operator detection processor comprising;

a first processor portion capable of averaging the first N consecutive correlation coefficients generated to generate a first average correlation coefficient, wherein N corresponds to at least the number of images required to image a blink of the operator'"'"'s eye,

a second processor portion capable of, after the production of the next image by the imaging apparatus, averaging the previous N consecutive correlation coefficients generated to create a next average correlation coefficient, and repeating the averaging for each image frame produced by the imaging apparatus,

a third processor portion capable of analyzing said average correlation coefficients to extract at least one parameter attributable to an eyeblink of said operator'"'"'s eye,

a fourth processor portion capable of comparing each extracted parameter to an alert operator threshold associated with that parameter, said threshold being indicative of an alert operator; and

an impaired operator warning unit comprising an impaired operator warning processor, said impaired operator warning processor capable of indicating that the operator may be impaired if any extracted parameter deviates from the associated threshold in a prescribed way.

17. The system of claim 16, wherein:

the third processor portion of the impaired operator detection processor is capable of extracting a parameter indicative of the duration of an operator'"'"'s eyeblinks;

the fourth processor portion of the impaired operator detection processor is capable of comparing the extracted duration parameter to said associated alert operator threshold wherein the threshold corresponds to a maximum eyeblink duration expected to be exhibited by an alert operator'"'"'s eye; and

the impaired operator warning processor is capable of indicating the operator may be impaired whenever the extracted duration parameter exceeds the alert operator duration threshold.

18. The system of claim 16, wherein:

the third processor portion of the impaired operator detection processor is capable of extracting a parameter indicative of the frequency of an operator'"'"'s eyeblinks;

the fourth processor portion of the impaired operator detection processor is capable of comparing the extracted frequency parameter to said associated alert operator threshold wherein the threshold corresponds to a minimum eyeblink frequency expected to be exhibited by an alert operator'"'"'s eye; and

the impaired operator warning processor is capable of indicating the operator may be impaired whenever the extracted frequency parameter is less than the alert operator frequency threshold.

19. The system of claim 16, wherein:

the third processor portion of the impaired operator detection processor is capable of extracting a parameter indicative of the amplitude of an operator'"'"'s eyeblinks;

the fourth processor portion of the impaired operator detection processor is capable of comparing the extracted amplitude parameter to said associated alert operator threshold wherein the threshold corresponds to a minimum eyeblink amplitude expected to be exhibited by an alert operator'"'"'s eye; and

the impaired operator warning processor is capable of indicating the operator may be impaired whenever the extracted amplitude parameter is less than the alert operator amplitude threshold.

20. The system of claim 16, wherein:

the first processor portion of the eye finding processor is further capable of determining the location of an other one of the operator'"'"'s eyes within each digital image;

the second processor portion of the eye finding processor is further capable of generating correlation coefficients each of which quantifies the degree of correspondence between pixels associated with the location of the operator'"'"'s other eye in an immediately preceding image in comparison to pixels associated with the location of the operator'"'"'s other eye in a current image;

the first processor portion of the impaired operator detection processor is further capable of averaging the first N consecutive correlation coefficients generated to generate a first average correlation coefficient associated with the operator'"'"'s other eye, wherein N corresponds to at least the number of images required to image a blink of the operator'"'"'s other eye,

the second processor portion of the impaired operator detection processor is further capable of, after the production of the next image by the imaging apparatus, averaging the previous N consecutive correlation coefficients generated and associated with the operator'"'"'s other eye, to create a next average correlation coefficient associated with the operator'"'"'s other eye, and repeating the averaging for each image frame produced by the imaging apparatus,

the third processor portion of the impaired operator detection processor is further capable of analyzing said average correlation coefficients associated with the operator'"'"'s other eye to extract at least one parameter attributable to an eyeblink of said operator'"'"'s other eye,

the fourth processor portion of the impaired operator detection processor is further capable of comparing each extracted parameter associated with the operator'"'"'s other eye to an alert operator threshold associated with that parameter, said threshold being indicative of an alert operator;

the impaired operator warning processor is further capable of indicating that the operator may be impaired if any extracted parameter associated with the operator'"'"'s other eye deviates from the associated threshold in a prescribed way.

21. The system of claim 20, wherein

the fourth processor portion of the impaired operator detection processor is further capable of first determining the difference between an extracted parameter associated with the operator'"'"'s first eye and a like extracted parameter associated with the operator'"'"'s other eye to establish a parameter consistency factor for the extracted parameter, and thereafter comparing the established parameter consistency factor to an alert operator consistency threshold associated with that parameter.

22. The system of claim 21, wherein:

the third processor portion of the impaired operator detection processor is further capable of extracting a parameter indicative of the duration of an operator'"'"'s eyeblinks;

the fourth processor portion of the impaired operator detection processor is further capable of determining the difference between the duration of each of the operator'"'"'s eyeblinks associated with the first eye to the duration of a next occurring eyeblink associated with the other eye to establish a duration consistency factor, and thereafter comparing the determined duration consistency factor to said associated alert operator consistency threshold wherein the threshold corresponds to a minimum difference in the eyeblink duration expected to be exhibited by an alert operator'"'"'s eyes; and

the impaired operator warning processor is further capable of indicating that the operator may be impaired whenever the determined duration consistency factor exceeds the associated alert operator duration consistency threshold.

23. The system of claim 21, wherein:

the third processor portion of the impaired operator detection processor is further capable of extracting parameters indicative of the frequency of an operator'"'"'s eyeblinks;

the fourth processor portion of the impaired operator detection processor is further capable of determining the difference between the frequency of the operator'"'"'s eyeblinks associated with the first eye as calculated over a prescribed period of time to the contemporaneous frequency of the eyeblinks associated with the other eye as calculated for the prescribed period of time to establish a frequency consistency factor, and thereafter comparing the determined frequency consistency factor to said associated alert operator consistency threshold wherein the threshold corresponds to a minimum difference in the eyeblink frequency expected to be exhibited by an alert operator'"'"'s eyes; and

the impaired operator warning processor is further capable of indicating that the operator may be impaired whenever the determined frequency consistency factor exceeds the alert operator frequency consistency threshold.

24. The system of claim 21, wherein:

the third processor portion of the impaired operator detection processor is further capable of extracting parameters indicative of the amplitude of an operator'"'"'s eyeblinks;

the fourth processor portion of the impaired operator detection processor is further capable of determining the difference between the amplitude of each of the operator'"'"'s eyeblinks associated with the first eye to the completeness of the next occurring eyeblink associated with the other eye to establish a amplitude consistency factor, and thereafter comparing the determined amplitude consistency factor to said associated alert operator consistency threshold wherein the threshold corresponds to a minimum difference in the amplitude of eyeblinks expected to be exhibited by an alert operator'"'"'s eyes; and

the impaired operator warning processor is further capable of indicating that the operator may be impaired whenever the determined amplitude consistency factor exceeds the alert operator amplitude consistency threshold.

25. The system of claim 20, wherein plural parameters attributable to an eyeblink of said operator'"'"'s eyes are extracted, and wherein:

the third processor portion of the impaired operator detection processor is further capable of extracting parameters indicative of the duration, frequency, and amplitude of an operator'"'"'s eyeblinks;

the fourth processor portion of the impaired operator detection processor is further capable of;

comparing each extracted duration parameter associated with the operator'"'"'s eyes to an alert operator duration threshold wherein the duration threshold corresponds to a maximum eyeblink duration expected to be exhibited by an alert operator'"'"'s eyes,

comparing each extracted frequency parameter associated with the operator'"'"'s eyes to an alert operator frequency threshold wherein the frequency threshold corresponds to a minimum eyeblink frequency expected to be exhibited by an alert operator'"'"'s eyes,

comparing each extracted amplitude parameter associated with the operator'"'"'s eyes to an alert operator amplitude threshold wherein the amplitude corresponds to a minimum eyeblink amplitude expected to be exhibited by an alert operator'"'"'s eyes, and

determining the difference between at least one of the extracted parameters associated with the first eye and a like extracted parameter associated with the other eye to establish a consistency factor for the extracted parameter, and thereafter comparing the established parameter consistency factor to an alert operator consistency threshold associated with that parameter.

26. The system of claim 25, wherein the impaired operator warning processor is further capable of indicating the operator may be impaired whenever at least one of (1) any extracted duration parameter exceeds the alert operator duration threshold, (2) any extracted frequency parameter is less than the alert operator frequency threshold, (3) any extracted amplitude parameter is less than the alert operator amplitude threshold, and (4) an established parameter consistency factor exceeds an associated alert operator consistency threshold.

27. The system of claim 16, further comprising:

a corroborating operator alertness indicator unit capable of generating a corroborating indicator of operator impairedness whenever operator control inputs are indicative of the operator being impaired; and

wherein

the impaired operator warning processor indicates that the operator is impaired whenever at least one of the extracted parameter deviates from the associated threshold in a prescribed way, and the corroborating indicator is generated.