Algorithm for monitoring head/eye motion for driver alertness with one camera

• US 6,927,694 B1
• Filed: 08/16/2002
• Issued: 08/09/2005
• Est. Priority Date: 08/20/2001
• Status: Expired due to Term

• Current Assignee: University of Central Florida (State University System of Florida)
• Original Assignee: University of Central Florida Research Foundation Inc. (State University System of Florida)
• Inventors: Lobo, Niels da Vitoria, Shah, Mubarak, Smith, Raymond Paul
• Primary Examiner(s): Nguyen, Phung
• Application Number:
  US10/222,432
• Time in Patent Office: 1,089 Days
• Field of Search: 340/576, 340/573.1, 340/575, 340/937, 340/439, 340/425.5, 180/272, 180/271, 382/117, 348/77, 348/78
• US Class Current: 340/576
• CPC Class Codes: B60K 28/066 actuating a signalling device B60K28/063 takes precedence
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms

Abstract

Visual methods and systems are described for detecting alertness and vigilance of persons under conditions of fatigue, lack of sleep, and exposure to mind altering substances such as alcohol and drugs. In particular, the intention can have particular applications for truck drivers, bus drivers, train operators, pilots and watercraft controllers and stationary heavy equipment operators, and students and employees during either daytime or nighttime conditions. The invention robustly tracks a person'"'"'s head and facial features with a single on-board camera with a fully automatic system, that can initialize automatically, and can reinitialize when it need'"'"'s to and provide outputs in realtime. The system can classify rotation in all viewing direction, detects'"'"' eye/mouth occlusion, detects'"'"' eye blinking, and recovers the 3D(three dimensional) gaze of the eyes. In addition, the system is able to track both through occlusion like eye blinking and also through occlusion like rotation. Outputs can be visual and sound alarms to the driver directly. Additional outputs can slow down the vehicle cause and/or cause the vehicle to come to a full stop. Further outputs can send data on driver, operator, student and employee vigilance to remote locales as needed for alarms and initiating other actions.

Claims

1. A driver alertness system comprising:

(a) a single camera within a vehicle aimed at a head region of a driver;

(b) means for simultaneously monitoring head rotation, yawning and full eye occlusion of the driver with said camera, the head rotation including nodding up and down, and moving left to right, and the full eye occlusion including eye blinking and complete eye closure, the monitoring means includes means for determining left to right rotation and the up and down nodding from examining approximately 10 frames from approximately 20 frames; and

(c) alarm means for activating an alarm in real time when a threshold condition in the monitoring means has been reached, whereby the driver is alerted into driver vigilance.

2. The system of claim 1 whereby the monitoring means includes:

means for determining gaze direction of the driver.

3. The system of claim 1, wherein the monitoring means includes:

a detected condition selected from at least one of;

lack of sleep of the driver, driver fatigue, alcohol effects and drug effects of the driver.

4. The system of claim 1, wherein the monitoring means includes:

inializing means to find a face of the driver;

grabbing means to grab a frame;

tracking means to truck head of the driver;

measuring means to measure rotation and nodding of the driver;

detecting means to detect eye blinking and eye closures of the driver;

yawing means to detect yawning of the driver.

5. A method of detecting driver vigilance, comprising the steps of:

aiming a single camera at a head of a driver of a vehicle;

detecting frequency of up and down nodding and left to right rotations of the head within a selected time period of the driver with the camera;

determining frequency of eye blinkings and eye closings of the driver within the selected time period with the camera;

determining the left to right head rotations and the up and down head nodding from examining approximately 10 frames from approximately 20 frames;

detecting frequency of yawning of the driver within the selected time period with the camera; and

generating an alarm signal in real time if a frequency value of the number of the frequency of the up and down nodding, the left to right rotations, the eye blinkings, the eye closings, the yawning exceed a selected threshold value.

6. The method of claim 5, further comprising the step of:

determining day and nighttime conditions, and generating the alarm signals in real time during both the day and the nighttime conditions.

7. The method of claim 5, wherein the selected time period includes:

at least approximately 20 frames.

8. The method of claim 5, wherein the selected time period includes:

within approximately 2 seconds.

9. The method of claim 5, wherein the generating step further includes the step of:

generating an onboard alarm to the driver in realtime.

10. The method of claim 5, wherein the generating step further includes the step of:

slowly slowing down the vehicle to a full stop.

11. The method of claim 5, wherein the generating step further includes the step of:

remotely transmitting the alarm signal to a third party located at a different location than the driver.

12. The method of claim 5, further comprising the step of:

providing the vehicle be selected from one of;

a cab, a bus and a truck; and

providing a dispatched at the different location.

13. The method of claim 5, further comprising the step of:

providing the vehicle be a plane; and

providing a control tower as the different location.

14. The method of claim 5, further comprising the step of:

providing the vehicle be a watercraft; and

providing a harbor master at the different location.

15. The method of claim 5, further comprising the step of:

providing the vehicle be a train; and

providing a station master at the different location.

16. The method of claim 5, further comprising the step of:

detecting at least one of driver fatigue, driver lack of sleep, driver alcohol intoxication, and driver drug intoxication.

17. The method of claim 5, further comprising the steps of:

inializing to find a face of the driver;

grabbing a frame;

tracking the head of the driver;

measuring the rotation and the nodding of the driver;

detecting the eye blinking and the eye closures of the driver;

detecting the yawning of the driver.

18. A vigilance monitoring system, comprising:

means for detecting up and down head nodding, and left to right head rotation of a person to be monitored, the detecting means including a means for determining the left to right head rotation and the up and down nodding from examining approximately 10 frames from approximately 20 frames;

means for detecting eye blinkings and complete eye closures of the person;

means for detecting yawning of the person; and

means for simultaneously monitoring the up and down head nodding and the head rotation detecting means, the eye blinking and the closure detecting means and the yawning detecting means within one computer.

19. The vigilance monitoring system of claim 18, further comprising:

a single camera for receiving visual images for the head rotation detecting means, the eye closure detecting means and the yawning detecting means.

20. The vigilance monitoring system of claim 18, wherein the person is a driver in a vehicle.

21. The vigilance monitoring system of claim 18 wherein the person is a pilot in a plane.

22. The vigilance monitoring system of claim 18, wherein the person is a skipper in a watercraft.

23. The vigilance monitoring system of claim 18, wherein the person is a driver of a truck.

24. The vigilance monitoring system of claim 18, wherein the person is an operator of heavy stationary equipment.

25. The vigilance monitoring system of claim 18, wherein the person is a student attending a class.

26. The vigilance monitoring system of claim 18, wherein the person is an employee at a workstation.

27. The vigilance monitoring system of claim 18, wherein the person is an employee being trained.

28. The vigilance monitoring system of claim 18, further comprising:

means for outputting an alarm signal when a selected threshold is reached from at least one of;

the head nodding and the rotation detecting means, the yawning detecting means and the eye blinking and the eye closure detection means.

29. The vigilance monitoring system of claim 18, further comprising:

means for monitoring the person during both night time conditions.

30. The system of claim 18, wherein the monitoring means includes:

inializing means to find face of the driver;

grabbing means to grab a frame;

tracking means to track head of the driver;

measuring means to measure rotation and nodding of the driver;

detecting means to detect eye blinking and eye closures of the driver;

yawing means to detect yawning of the driver.

31. A method of monitoring attention vigilance, comprising the steps of:

detecting head nodding and head rotation of a person to be monitored;

determining left to right rotation and up and down nodding from examining approximately 10 frames from approximately 20 frames;

detecting eye blinking and eye closures of the person;

detecting yawning of the person; and

simultaneously monitoring the head nodding and the head rotation, the eye blinking and eye closures and the yawning within one computer.

32. The method of claim 31, further comprising the step of:

detecting the nodding and the head rotation, the eye blinking and the closures and the yawning with a single camera.

33. The method of claim 31, further comprising the step of:

monitoring the person with identical equipment during day and night time conditions.

34. The method of claim 31, further comprising the step of:

detecting attention vigilance of a driver of a vehicle.

35. The method of claim 31, further comprising the step of:

detecting attention vigilance of an operator of stationary equipment.

36. The method of claim 31, further comprising the step of:

detecting attention vigilance of a pilot of a plane.

37. The method of claim 31, further comprising the step of:

detecting attention vigilance of an operator of a watercraft.

38. The method of claim 31, further comprising the step of:

detecting attention vigilance of a student being educated.

39. The method of claim 31, further comprising the step of:

detecting attention vigilance of an employee at work.

40. The method of claim 31, further comprising the steps of:

inializing to find face of the driver;

grabbing a frame;

tracking the head of the driver;

measuring the rotation and the nodding of the driver;

detecting the eye blinking and the eye closures of the driver;

detecting the yawning of the driver.