AN AUTOMATED METHOD FOR HUMAN FACE MODELING AND RELIGHTING WITH APPLICATION TO FACE RECOGNITION

• US 20090310828A1
• Filed: 10/13/2008
• Published: 12/17/2009
• Est. Priority Date: 10/12/2007
• Status: Active Grant

• Current Assignee: University of Houston System
• Original Assignee: University of Houston System
• Inventors: Toderici, George, Passalis, Georgios, Kakadiaris, Ioannis A., Theoharis, Theoharis
• Granted Patent: US 8,090,160 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A novel method and system for 3d-aided-2D face recognition under large pose and illumination variations is disclosed. The method and system includes enrolling a face of a subject into a gallery database using raw 3D data. The method also includes verifying and/or identifying a target face form data produced by a 2D imagining or scanning device. A statistically derived annotated face model is fitted using a subdivision-based deformable model framework to the raw 3D data. The annotated face model is capable of being smoothly deformed into any face so it acts as a universal facial template. During authentication or identification, only a single 2D image is required. The subject specific fitted annotated face model from the gallery is used to lift a texture of a face from a 2D probe image, and a bidirectional relighting algorithm is employed to change the illumination of the gallery texture to match that of the probe. Then, the relit texture is compared to the gallery texture using a view-dependent complex wavelet structural similarity index metric.

Claims

1. A method for 3D-aided 2D face recognition under large pose and illumination variations implemented on a computer or on a distributed computer network, where the method comprises:

enrolling a face of a subject into a gallery database as a fitted annotated face model, where the enrollment data comprises raw 3D data obtained either from a 3D scanner or imaging apparatus or derived from a 2D imaging or scanning apparatus and where a statistically derived annotated face model is fitted using a subdivision-based deformable model framework to the raw 3D data,

obtaining raw 2D data of a target face from a 2D imagining or scanning apparatus or probe,

lifting a texture of the target face from the 2D probe image,

selecting a specific fitted annotated face model from the gallery for an authentication experiment or all the gallery models for an identification experiment,

bidirectionally relighting the gallery texture from the selected specific fitted annotated face model to change an illumination of the gallery texture to match the texture lifted from the probe image,

comparing the two matched textures, and

verifying and/or identifying the target face based on a value of a metric.

2. The method of claim 1, wherein the enrolling step comprises:

obtaining and storing on a computer, raw 3D data of a face of a subject to be enrolled either directly from a 3D imaging apparatus or derived from a 2D imaging apparatus,

fitting an annotated face model stored in the computer onto the raw 3D data using a subdivision-based deformable model framework using routines encoded on the computer to form a fitted face model,

converting the fitted face model using routines encoded on the computer to a geometry image representation comprising regularly sampled, registered and annotated geometry data,

lifting an albedo of the skin from the stored raw 3D data using an analytical skin reflectance model stored in the computer, where the analytical skin reflectance model is adapted to remove lighting, shadows, diffuse light and specular light from the albedo texture to form a lighting invariant albedo texture of the skin,

placing the lighting invariant albedo texture onto the geometry image representation to form a subject specific deformable and relightable construct, and

storing the construct in a gallery database encoded on the computer, where the database include constructs for all enrolled subjects,

where the method requires no measurement or calibration of a lighting environment, is fully automatic, has minimum input requirements making it applicable to both computer vision applications and computer graphics applications and allowing utilization of existing 3D facial data stored in other databases.

3. The method of claim 2, wherein computer vision applications include face recognition and face verification systems.

4. The method of claim 2, wherein computer graphic application include face relighting, face synthesis and facial expressions transference.

5. The method of claim 2, further comprising:

if the raw 3D data includes over-saturated specular highlights, inpainting the albedo texture to produce an estimated the lighting invariant albedo texture.

6. A method for 3D-aided 2D face recognition under large pose and illumination variations implemented on a computer or on a distributed computer network, where the method comprises:

obtaining and storing on a computer, raw 3D data of a face of a subject to be enrolled either directly from a 3D imaging apparatus or derived from a 2D imaging apparatus,

fitting an annotated face model stored in the computer onto the raw 3D data using a subdivision-based deformable model framework using routines encoded on the computer to form a fitted face model,

converting the fitted face model using routines encoded on the computer to a geometry image representation comprising regularly sampled, registered and annotated geometry data,

lifting an albedo of the skin from the stored raw 3D data using an analytical skin reflectance model stored in the computer, where the analytical skin reflectance model is adapted to remove lighting, shadows, diffuse light and specular light from the albedo texture to form a lighting invariant albedo texture of the skin,

placing the lighting invariant albedo texture onto the geometry image representation to form a subject specific deformable and relightable construct, and

storing the construct in a gallery database encoded on the computer, where the database include constructs for all enrolled subjects,

where the method requires no measurement or calibration of a lighting environment, is fully automatic, has minimum input requirements making it applicable to both computer vision applications and computer graphics applications and allowing utilization of existing 3D facial data stored in other databases,

obtaining raw 2D data of a target face from a 2D imagining or scanning apparatus or probe,

lifting a texture of the target face from the 2D probe image,

selecting a specific fitted annotated face model from the gallery for an authentication experiment or all the gallery models for an identification experiment,

bidirectionally relighting the gallery texture from the selected specific fitted annotated face model to change an illumination of the gallery texture to match the texture lifted from the probe image,

comparing the two matched textures,

verifying and/or identifying the target face based on a value of a metric.

7. A method implemented on a computer or on a distributed computer network comprising:

obtaining and storing on a computer, raw 3D data of a face of a subject to be enrolled either directly from a 3D imaging apparatus or derived from a 2D imaging apparatus,

fitting an annotated face model stored in the computer onto the raw 3D data using a subdivision-based deformable model framework using routines encoded on the computer to form a fitted face model,

converting the fitted face model using routines encoded on the computer to a geometry image representation comprising regularly sampled, registered and annotated geometry data,

lifting an albedo of the skin from the stored raw 3D data using an analytical skin reflectance model stored in the computer, where the analytical skin reflectance model is adapted to remove lighting, shadows, diffuse light and specular light from the albedo texture to form a lighting invariant albedo texture of the skin,

placing the lighting invariant albedo texture onto the geometry image representation to form a subject specific deformable and relightable construct, and

storing the construct in a gallery database encoded on the computer, where the database include constructs for all enrolled subjects,

where the method requires no measurement or calibration of a lighting environment, is fully automatic, has minimum input requirements making it applicable to both computer vision applications and computer graphics applications and allowing utilization of existing 3D facial data stored in other databases.

8. The method of claim 7, wherein computer vision applications include face recognition and face verification systems.

9. The method of claim 7, wherein computer graphic application include face relighting, face synthesis and facial expressions transference.

10. The method of claim 7, further comprising:

if the raw 3D data includes over-saturated specular highlights, inpainting the albedo texture to produce an estimated the lighting invariant albedo texture.

11. The method of claim 7, further comprising:

deforming a facial expression from a first construct in the gallery database to match a second construction in the gallery and

transferring the facial expression from the first construct to the second construct to form a synthetic facial construct.

12. The method of claim 7, further comprising:

retrieving a construct from the gallery database and

relighting the construct to form a relit construct, where the relighting is user defined.

13. The method of claim 7, further comprising:

retrieving a set of attributes from a plurality of constructs in the gallery database and

forming a synthetic construct from the set of attributes.

15. The system of claim 11, wherein the computer further having implemented thereon a notification method adapted to notify a user concerning the identity of a target subject.

14. A system comprising:

an enrollment station or a small plurality of enrollment stations adapted to produce raw 3D data of a subject to be enrolled in the system,

a computer in data communication with the enrollment stations having implemented thereon an enrollment method comprising;

obtaining and storing on a computer, raw 3D data of a face of a subject to be enrolled either directly from a 3D imaging apparatus or derived from a 2D imaging apparatus,

fitting an annotated face model stored in the computer onto the raw 3D data using a subdivision-based deformable model framework using routines encoded on the computer to form a fitted face model,

converting the fitted face model using routines encoded on the computer to a geometry image representation comprising regularly sampled, registered and annotated geometry data,

lifting an albedo of the skin from the stored raw 3D data using an analytical skin reflectance model stored in the computer, where the analytical skin reflectance model is adapted to remove lighting, shadows, diffuse light and specular light from the albedo texture to form a lighting invariant albedo texture of the skin,

placing the lighting invariant albedo texture onto the geometry image representation to form a subject specific deformable and relightable construct, and

storing the construct in a gallery database encoded on the computer, where the database include constructs for all enrolled subjects,

where the method requires no measurement or calibration of a lighting environment, is fully automatic, has minimum input requirements making it applicable to both computer vision applications and computer graphics applications and allowing utilization of existing 3D facial data stored in other databases,

a plurality of identification stations adapted to produce raw 2D facial data of a target subject or a plurality of target subjects in data communication with the computer, where the computer further having implemented thereon an identification method comprising;

lifting a texture of the target face from the 2D probe image,

selecting a specific fitted annotated face model from the gallery for an authentication experiment or all the gallery models for an identification experiment,

bidirectionally relighting the gallery texture from the selected specific fitted annotated face model to change an illumination of the gallery texture to match the texture lifted from the probe image,

comparing the two textures matched using a view-dependent complex wavelet structural similarity index metric,

verifying and/or identifying the target face based on a value of the metric,

where an equal error rate (EER) of this method is less than half of the best commercially available 2D face recognition method.