System for rapid detection of drowsiness in a machine operator

• US 8,040,247 B2
• Filed: 03/23/2009
• Issued: 10/18/2011
• Est. Priority Date: 03/23/2009
• Status: Active Grant

• Current Assignee: Toyota Motor Corporation
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Gunaratne, Pujitha
• Primary Examiner(s): Previl; Daniel
• Application Number:
  US12/409,140
• Publication Number:
  US 20100238034A1
• Time in Patent Office: 939 Days
• Field of Search: 340/575, 340/573.1, 340/576, 340/539.1, 340/426.19, 340/989, 340/991, 340/539.11, 340/5.1, 340 581- 583, 340/4.1
• US Class Current: 340/575
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: B60K 28/066 actuating a signalling device B60K28/063 takes precedence
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions

Abstract

The present invention discloses a system and a process for rapidly detecting drowsiness in a individual operating a machine. The system can capture a plurality of facial images of the individual and compare one or more facial parameters from the images to a plurality of stored high priority sleepiness facial actions that are in a prioritized action database. Based on the comparison, a current level of sleepiness can be determined for the individual and an actuator can be actuated in order to alert the individual and possibly any other individuals that may be in the vicinity.

Claims

1. A system for detecting drowsiness in a machine operator, the system comprising:

an image acquisition module operable to electronically capture a plurality of facial images of the machine operator;

a face localization module having a generic face mesh grid, said face localization module operable to create a neutral face mesh grid by fitting said generic face mesh grid to a desired first captured facial image, said face localization module also operable to create a subsequent face mesh grid by fitting said generic face mesh grid to a desired second captured facial image;

a facial orientation module operable to determine a facial variation between a facial orientation parameter of said neutral face mesh grid and a facial orientation parameter of said subsequent face mesh grid;

a facial action database having a plurality of stored high priority sleepiness facial actions, said plurality of stored high priority sleepiness facial actions each having an assigned sleepiness severity value;

an action estimation module having a comparison algorithm, said action estimation module operable to compare said facial variation with said plurality of stored high priority sleepiness facial actions using said comparison algorithm and select a stored high priority sleepiness facial action that matches said facial variation within a predetermined amount;

a matching module operable to identify and select stored high priority sleepiness facial actions that are within a predetermined range of likeness to said identified stored high priority sleepiness facial action;

a sleepiness level module operable to compare assigned sleepiness values of all selected stored high priority sleepiness facial actions and generate a probability of a current level of sleepiness;

a warning module operable to actuate an actuator as a function of the current level of sleepiness having a highest probability.

2. The system of claim 1, wherein said face localization module has a facial landmark point circuit operable to select a set of facial landmark points on each of said plurality of captured images.

3. The system of claim 2, wherein said face localization module uses said set of facial landmark points to fit said generic face mesh grid to said desired first captured facial image and said desired second captured facial image.

4. The system of claim 1, wherein said facial orientation parameter is selected from the group consisting of a pose parameter, an eye parameter, a mouth parameter and combinations thereof.

5. The system of claim 4, wherein said facial orientation module has a circuit selected from the group consisting of a pose estimation circuit, a facial parts segmentation circuit and combinations thereof.

6. The system of claim 5, wherein said pose estimation circuit is operable to calculate a pose variation between a pose parameter of said subsequent face mesh grid of the operator in three-dimensional space and a pose parameter of said neutral face mesh grid in three-dimensional space of the operator.

7. The system of claim 5, wherein said facial parts segmentation circuit has a boundary tracing circuit selected from the group consisting of an eye region tracing circuit, a mouth region tracing circuit and combinations thereof, said eye region tracing circuit operable to trace an eye region on a face mesh grid and create said eye parameter and said mouth region tracing circuit operable to trace a mouth region on said face mesh grid and can create said mouth parameter.

8. The system of claim 7, wherein said eye parameter is an eye region boundary and said mouth parameter is a mouth region boundary.

9. The system of claim 8, wherein said facial parts segmentation circuit is operable to calculate an eye region variation between an eye region boundary of said subsequent face mesh grid and an eye region boundary of said neutral face mesh grid of the operator.

10. The system of claim 9, wherein said facial parts segmentation circuit is operable to calculate a mouth region variation between a mouth region boundary of said subsequent face mesh grid and a mouth region boundary of said neutral face mesh grid of the machine operator.

11. The system of claim 1, further comprising a trigger data packet module operable to generate a trigger data packet containing data on all selected stored high priority sleepiness facial actions.

12. The system of claim 11, further comprising a drowsiness trigger module operable to evaluate said selected stored high priority sleepiness facial actions with said assigned sleepiness values with a previous probability of current sleepiness level and determine a highest probability candidate high priority sleepiness action.

13. The system of claim 12, wherein said drowsiness trigger module generates a drowsiness trigger as a function of said highest probability candidate high priority sleepiness action.

14. The system of claim 1, wherein said actuator is selected from the group consisting of an audio actuator, a light actuator, a tactile actuator and combinations thereof.

15. A process for detecting drowsiness in a machine operator, the process comprising:

capturing a plurality of facial images as a function of time the operator is operating the machine;

selecting a set of facial landmark points on a desired first captured image;

selecting a set of facial landmark points on a desired second captured image;

creating a neutral face mesh grid by fitting a generic face mesh grid to the set of facial landmark points on the desired first captured facial image;

creating a subsequent face mesh grid by fitting the generic face mesh grid the set of facial landmark points on the desired second captured facial image;

calculating a facial variation between the neutral face mesh grid and the subsequent facial mesh grid;

providing a facial action database having a plurality of stored high priority sleepiness facial actions, said plurality of stored high priority sleepiness facial actions each having an assigned sleepiness severity value;

comparing the facial variation to the plurality of stored high priority sleepiness facial actions;

selecting one of the stored high priority sleepiness facial actions that is within a predetermined range of likeness of the facial variation;

comparing the selected stored high priority sleepiness facial action with all of the plurality of stored high priority sleepiness facial actions;

selecting all of the stored high priority sleepiness facial actions that are similar to the selected one stored high priority sleepiness facial action within a predetermined range of likeness;

generating a trigger packet containing data on all of the selected stored high priority sleepiness facial actions;

evaluating all of the assigned sleepiness severity values for all of the selected stored high priority sleepiness facial actions;

calculating a probability for each of the selected stored high priority sleepiness facial actions;

calculating a current sleepiness level as a function of the selected stored high priority sleepiness facial action having a highest probability; and

actuating an actuator as a function of the current sleepiness level.

16. The process of claim 15, further including:

evaluating the current sleepiness level with a previous selected stored high priority sleepiness facial action having the highest probability;

calculating an updated current sleepiness level as a function of the evaluation of the current sleepiness level with the previous selected stored high priority sleepiness facial action having the highest probability.