Method and apparatus for detection of drowsiness

• US 6,717,518 B1
• Filed: 02/09/2001
• Issued: 04/06/2004
• Est. Priority Date: 01/15/1998
• Status: Expired due to Term

• Current Assignee: Image Processing Technologies, LLC (General Patent Corporation), William J. Brown
• Original Assignee: Holding B.E.V. S.A.
• Inventors: Binford, Thomas, Pirim, Patrick
• Primary Examiner(s): MULLEN, THOMAS J
• Application Number:
  US09/600,390
• Time in Patent Office: 1,152 Days
• Field of Search: 340/575, 340/576, 382/100, 382/103, 382/115, 382/117, 348/143, 348/148, 348/672
• US Class Current: 340/576
• CPC Class Codes: A61B 2503/22 Motor vehicles operators, e.g. drivers, pilots, captains
• CPC Class Codes: A61B 5/1103 Detecting eye twinkling
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: A61B 5/6821 Eye
• CPC Class Codes: A61B 5/6893 Cars
• CPC Class Codes: B60R 1/04 mounted inside vehicle B60R1/025, B60R1/08 take precedence
• CPC Class Codes: B60R 1/12 Mirror assemblies combined with other articles, e.g. clocks
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms

Abstract

In a process of detecting a person falling asleep, an image of the face of the person is acquired. Pixels of the image having characteristics corresponding to an eye of the person are selected and a histogram is formed of the selected pixels. The histogram is analyzed over time to identify each opening and closing of the eye, and characteristics indicative of the person falling asleep are determined. A sub-area of the image including the eye may be determined by identifying the head or a facial characteristic of the person, and then identifying the sub-area using an anthropomorphic model. To determine openings and closings of the eyes, histograms of shadowed pixels of the eye are analyzed to determine the width and height of the shadowing, or histograms of movement corresponding to blinking are analyzed. An apparatus for detecting a person falling asleep includes a sensor for acquiring an image of the face of the person, a controller, and a histogram formation unit for forming a histogram on pixels having selected characteristics. Also disclosed is a rear-view mirror assembly incorporating the apparatus.

Claims

1. A process of detecting a person falling asleep, the process comprising:

acquiring an image of the face of the person;

identifying a sub-area of the image comprising at least one eye of the person, including;

identifying the head of the person in the image; and

identifying the sub-area of the image using an anthropomorphic model;

selecting pixels within the sub-area of the image having characteristics corresponding to characteristics of the at least one eye of the person;

forming at least one histogram of the selected pixels;

analyzing the at least one histogram over time to identify each opening and closing of the eye; and

determining from the opening and closing information of the eye, characteristics indicative of a person falling asleep.

2. The process according to claim 1 wherein identifying the head of the person in the image comprises:

selecting pixels of the image having characteristics corresponding to edges of the head of the person;

forming histograms of the selected pixels projected onto forthogonal axes; and

analyzing the histograms of the selected pixels to identify the edges of the head of the person.

3. A process of detecting a person falling asleep, the process comprising:

acquiring an image of the face of the person;

identifying a sub-area of the image comprising at least one eye of the person, including;

identifying the location of a facial characteristic of the person in the image; and

identifying the sub-area of the image using an anthropomorphic model and the location of the facial characteristic;

selecting pixels within the sub-area of the image having characteristics corresponding to characteristics of the at least one eye of the person;

forming at least one histogram of the selected pixels;

analyzing the at least one histogram over time to identify each opening and closing of the eye; and

determining from the opening and closing information of the eye, characteristics indicative of a person falling asleep.

4. The process according to claim 3 wherein the step of identify the location of a facial characteristic of the person in the image comprises the steps of:

selecting pixels of the image having characteristics corresponding to the facial characteristic;

forming histograms of the selected pixels projected onto orthogonal axes; and

analyzing the histograms of the selected pixels to identify the position of the facial characteristic in the image.

5. The process according to claim 4 wherein the facial characteristic is the nostrils of the person, and wherein the step of selecting pixels of the image having characteristics corresponding to the facial characteristic comprises selecting pixels having low luminance levels.

6. The process according to claim 5 further comprising the step of analyzing the histograms of the nostril pixels to determine whether the spacing between the nostrils is within a desired range and whether the dimensions of the nostrils fall within a desired range.

7. The process according to claim 5 wherein the step of identifying the location of a facial characteristic of the person in the image comprises the step of searching sub-images of the image to identify the nostrils.

8. The process according to claim 3 wherein the step of identifying the location of a facial characteristic of the person in the image comprises the step of searching sub-images of the image to identify the facial characteristic.

9. The process according to claim 8 wherein the facial characteristic is a first facial characteristic and further comprising the steps of:

using an anthropomorphic model and the location of the first facial characteristic to select a sub-area of the image containing a second facial characteristic;

selecting pixels of the image having characteristics corresponding to the second facial characteristic; and

analyzing the histograms of the selected pixels of the second facial characteristic to confirm the identification of the first facial characteristic.

10. A process of detecting a person falling asleep, the process comprising the steps of

acquiring an image of the face of the person;

selecting pixels of the image having characteristics corresponding to characteristics of at least one eye of the person;

forming at least one histogram of the selected pixels;

analyzing the at least one histogram over time to identify each opening and closing of the eye; and

determining from the opening and closing information of the eye, characteristics indicative of a person falling asleep;

wherein the step of selecting pixels of the image comprises selecting pixels having low luminance levels corresponding to shadowing of the eye; and

wherein the step of analyzing the at least one histogram comprises analyzing the shape of the eye shadowing to determine openings and closings of the eye.

11. The process according to claim 10 wherein the step of forming at least one histogram of the selected pixels comprises forming histograms of shadowed pixels of the eye projected onto orthogonal axes, and wherein the step of analyzing the shape of the eye shadowing comprises analyzing the width and height of the shadowing.

12. A process of detecting a person falling asleep, the process comprising the steps of:

acquiring an image of the face of the person;

selecting pixels of the image having characteristics corresponding to characteristics of at least one eye of the person;

forming at least one histogram of the selected pixels;

analyzing the at least one histogram over time to identify each opening and closing of the eye; and

determining from the opening and closing information of the eye, characteristics indicative of a person falling asleep;

wherein the step of selecting pixels of the image comprises selecting pixels in movement corresponding to blinking; and

wherein the step of analyzing the at least one histogram comprises analyzing the number of pixels in movement over time to determine openings and closings of the eye.

13. The process according to claim 12 wherein the step of selecting pixels of the image having characteristics corresponding to characteristics of at least one eye of the person comprises selecting pixels having one or more characteristics selected from the group consisting of i) parameter DP=1 indicative of significant variation, ii) time constant CO indicative of a blinking eyelid, iii) velocity indicative of a blinking eyelid, and iv) up and down movement indicative of a blinking eyelid.

14. An apparatus for detecting a person falling asleep, the apparatus comprising:

a sensor for acquiring an image of the face of the person, the image comprising pixels corresponding to the eye of the person;

a controller; and

a histogram formation unit for forming a histogram on pixels having selected characteristics,

the controller controlling the histogram formation unit to select pixels of the image having characteristics corresponding to characteristics of at least one eye of the person and to for a histogram of the selected pixels, the controller analyzing the histogram over time to identify each opening and closing of the eye, and determining from the opening and closing information on the eye, characteristics indicative of a person falling asleep;

the controller interacting with the histogram formation unit to identify a sub-area of the image comprising the at least one eye, and the controller controls the histogram formation unit to select pixels of the image having characteristics corresponding to characteristics of at least one eye only within the sub-area of the image;

the controller interacting with the histogram formation unit to identify the head of the person in the image; and

the controller identifies the sub-area of the image using an anthropomorphic model.

15. The apparatus according to claim 14 wherein:

the histogram formation unit selects pixels of the image having characteristics corresponding to edges of the head of the person and forms histograms of the selected pixels projected onto orthogonal axes; and

the controller analyzes the histograms of the selected pixels to identify the edges of the head of the person.

16. The apparatus according to claim 14 wherein:

the controller interacts with the histogram formation unit to identify the location of a facial characteristic of the person in the image; and

the controller identifies the sub-area of the image using an anthropomorphic model and the location of the facial characteristic.

17. The apparatus according to claim 16 wherein:

the histogram formation unit selects pixels of the image having characteristics corresponding to the facial characteristic and forms histograms of the selected pixels projected onto orthogonal axes;

the controller analyzes the histograms of the selected pixels to identify the position of the facial characteristic in the image.

18. The apparatus according to claim 17 wherein the facial characteristic is the nostrils of the person, and wherein the histogram formation unit selects pixels of the image having low luminance levels corresponding to the luminance level of the nostrils.

19. The apparatus according to claim 18 wherein the controller analyzes the histograms of the nostril pixels to determine whether the spacing between the nostrils is within a desired range and whether the dimensions of the nostrils fall within a desired range.

20. The apparatus according to claim 18 wherein the controller interacts with the histogram formation unit to search sub-images of the image to identify the nostrils.

21. The apparatus according to claim 16 wherein the controller interacts with the histogram formation unit to search sub-images of the image to identify the facial characteristic.

22. The apparatus according to claim 21 wherein the facial characteristic is a first facial characteristic and further comprising:

the controller using an anthropomorphic model and the location of the first facial characteristic to cause the histogram formation unit to select a sub-area of the image containing a second facial characteristic, the histogram formation unit selecting pixels of the image in the sub-area having characteristics corresponding to the second facial characteristic and forming a histogram of such pixels; and

the controller analyzing the histogram of the selected pixels corresponding to the second facial characteristic to confirm the identification of the first facial characteristic.

23. The apparatus according to claim 14 wherein:

the histogram formation unit selects pixels of the image having low luminance levels corresponding to shadowing of the eye; and

wherein the controller analyzes the shape of the eye shadowing to determine openings and closings of the eye.

24. The apparatus according to claim 23 wherein histogram formation unit forms histograms of shadowed pixels of the eye projected onto orthogonal axes, and wherein the controller analyzes the width and height of the shadowing to determine openings and closings of the eye.

25. The apparatus according to claim 14 wherein:

the histogram formation unit selects pixels of the image in movement corresponding to blinking; and

the controller analyzes the number of pixels in movement over time to determine openings and closings of the eye.

26. The apparatus according to claim 25 wherein the histogram formation units selects pixels of the image having characteristics of movement corresponding to blinking, such characteristics being selected from the group consisting of i) parameter DP=1 indicative of significant variation, ii) time constant CO indicative of a blinking eyelid, iii) velocity indicative of a blinking eyelid, and iv) up and down movement indicative of a blinking eyelid.

27. The apparatus according to claim 14 wherein the sensor is integrally constructed with the controller and the histogram formation unit.

28. The apparatus according to claim 14 further comprising an alarm, the controller operating the alarm upon detection of the person falling asleep.

29. The apparatus according to claim 14 further comprising an illumination source, the sensor being adapted to view the person when illuminated by the illumination source.

30. The apparatus according to claim 29 wherein the illumination source is a source of IR radiation.

31. A rear-view mirror assembly for a vehicle which comprises:

a rear-view mirror; and

the apparatus according to claim 14 mounted to the rear-view mirror.

32. The rear-view mirror assembly according to claim 31 further comprising a bracket attaching the apparatus to the rear-view mirror.

33. The rear-view mirror assembly according to claim 31 further comprising a housing having an open side and an interior, the rear-view mirror being mounted to the open side of the housing, the rear view mirror being see-through from the interior of the housing to an exterior of the housing, the apparatus being mounted interior to the housing with the sensor directed toward the rear-view mirror.

34. The rear-view mirror assembly according to claim 33 further comprising a joint attaching the apparatus to the rear-view mirror assembly, the joint adapted to maintain the apparatus in a position facing a driver of the vehicle during adjustment of the mirror assembly by the driver.

35. The rear-view mirror assembly according to claim 31 further comprising a source of illumination directed toward the person, the sensor being adapted to view the person when illuminated by the source of illumination.

36. The rear-view mirror assembly according to claim 31 further comprising an alarm, the controller operating the alarm upon detection of the person falling asleep.

37. A rear-view mirror assembly which comprises:

a rear-view mirror; and

the apparatus according to claim 14, the sensor being mounted to the rear-view mirror, the controller and the histogram formation unit being located remote from the sensor.

38. A vehicle comprising the apparatus according to claim 14.

39. A process of detecting a feature of an eye, the process comprising the steps of:

acquiring an image of the face of the person, the image comprising pixels corresponding to the feature to be detected;

identifying a characteristic of the face other than the feature to be detected;

identifying a portion of the image of the face comprising the feature to be detected using an anthropomorphic model based on the location of the identified facial characteristic;

selecting pixels of the portion of the image having characteristics corresponding to the feature to be detected;

forming at least one histogram of the selected pixels; and

analyzing the at least one histogram over time to identify characteristics of the feature to be detected;

said feature being the iris, pupil or cornea.