System and method for detecting drowsy facial expressions of vehicle drivers under changing illumination conditions

• US 8,369,608 B2
• Filed: 06/22/2009
• Issued: 02/05/2013
• Est. Priority Date: 06/22/2009
• Status: Active Grant

• Current Assignee: Toyota Motor Corporation
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Gunaratne, Pujitha
• Primary Examiner(s): STREGE, JOHN B
• Application Number:
  US12/488,834
• Publication Number:
  US 20100322507A1
• Time in Patent Office: 1,324 Days
• Field of Search: 382/118, 382/154, 382/209, 382/224, 382/275
• US Class Current: 382/154
• CPC Class Codes: G06V 10/7557 based on appearance, e.g. active appearance models [AAM]
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

The present invention includes a method of detecting drowsy facial expressions of vehicle drivers under changing illumination conditions. The method includes capturing an image of a person'"'"'s face using an image sensor, detecting a face region of the image using a pattern classification algorithm, and performing, using an active appearance model algorithm, local pattern matching to identify a plurality of landmark points on the face region of the image. The method also includes generating a 3D face model with facial muscles of the face region, determining photometric flows from the 3D face model using an extract photometric flow module, determining geometric flows from the 3D face model using a compute geometric flow module, determining a noise component generated by varying illuminations by comparing the geometric flows to the photometric flows, and removing the noise component by subtracting two photometric flows.

Claims

1. A method of detecting drowsy facial expressions of vehicle drivers under changing illumination conditions comprising:

capturing an image of a person'"'"'s face using an image sensor;

detecting a face region of the image using a pattern classification algorithm;

performing, using an active appearance model algorithm, local pattern matching to identify a plurality of landmark points on the face region of the image;

generating a 3D face model with facial muscles of the face region;

determining photometric flows from the 3D face model using an extract photometric flow module;

determining geometric flows from the 3D face model using a compute geometric flow module;

determining a noise component generated by varying illuminations by comparing the geometric flows to the photometric flows; and

removing the noise component by subtracting two photometric flows.

2. The method of claim 1 wherein the image sensor is a charge-coupled device (CCD) or a complimentary metal oxide semiconductor (CMOS) active-pixel sensor.

3. The method of claim 1 wherein the plurality of landmark points is selected from a group consisting of an eye portion, a mouth portion, a nose tip portion, an ear-joint portion, and combinations thereof.

4. The method of claim 1 wherein the local pattern matching matches each of the plurality of landmark points with a set of stored facial values in a database.

5. The method of claim 1 wherein the generating a 3D face model with facial muscles of the face region includes:

obtaining a generic face model with pre-assigned facial muscles from a database; and

adapting the generic face model to the 3D face model with the facial muscles of the face region using a minimization algorithm.

6. The method of claim 4 wherein the set of stored facial values include a plurality of landmark markings from pre-annotated sets of images that were used as learning sets.

7. The method of claim 4 wherein the set of stored facial values include a spatial derivative between the plurality of landmark points.

8. The method of claim 4 wherein the set of stored facial values include a spatial derivative between the plurality of landmark points.

9. The method of claim 1 wherein the determined photometric flows define a motion flow between the facial muscles.

10. The method of claim 1 further comprising integrating the determined photometric flows with the facial muscles to generate a relationship between facial muscle movement in expressions and corresponding photometric flows.

11. The method of claim 1 further comprising outputting on a display a facial deformation and a corresponding facial expression.

12. A method of detecting facial expressions of a person under changing illumination conditions comprising:

capturing an image of at least a portion of the person'"'"'s face using an image sensor;

generating a model based on the image of at least the portion of the person'"'"'s face;

determining photometric flows based on the model using an extract photometric flow module;

determining geometric flows based on the model using a compute geometric flow module;

determining a noise component generated by varying illuminations by comparing the geometric flows to the photometric flows;

removing the noise component by subtracting two photometric flows; and

determining whether a deformation of the image of at least the portion of the person'"'"'s face exists based at least in part on the removal of the noise component.

13. The method of claim 12 further comprising outputting an audible alarm or a visual alarm if the deformation of the image of at least the portion of the person'"'"'s face exists.

14. The method of claim 12 wherein the step of generating a model includes generating a 3D face model with a facial muscle and further comprising the step of integrating the determined photometric flows with the facial muscle to generate a relationship between facial muscle movement in expressions and a corresponding photometric flow.

15. The method of claim 12 wherein the image sensor is a charge-coupled device (CCD) or a complimentary metal oxide semiconductor (CMOS) active-pixel sensor.

16. The method of claim 12 further comprising a step of performing, using an active appearance model algorithm, local pattern matching to identify a plurality of landmark points on the image of at least the portion of the person'"'"'s face.

17. The method of claim 16 wherein the plurality of landmark points is selected from a group consisting of an eye portion, a mouth portion, a nose tip portion, an ear joint portion, and combinations thereof.

18. The method of claim 16 wherein the local pattern matching matches each of the plurality of landmark points with a set of stored facial values in a database.

19. The method of claim 18 wherein the set of stored facial values include a plurality of landmark markings from pre-annotated sets of images that were used as learning sets.

20. The method of claim 12 wherein the generating a model based on the image of at least the portion of the person'"'"'s face includes:

obtaining a generic face model with pre-assigned facial muscles from a database; and

adapting the generic face model to the model to be generated with facial muscles of at least the portion of the person'"'"'s face using a minimization algorithm.