System and method for face recognition using synthesized training images

• US 20020106114A1
• Filed: 12/01/2000
• Published: 08/08/2002
• Est. Priority Date: 12/01/2000
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Yan, Jie, Zhang, Hong-Jiang
• Granted Patent: US 6,975,750 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A system and method that includes a virtual human face generation technique which synthesizes images of a human face at a variety of poses. This is preferably accomplished using just a frontal and profile image of a specific subject. An automatic deformation technique is used to align the features of a generic 3-D graphic face model with the corresponding features of these pre-provided images of the subject. Specifically, a generic frontal face model is aligned with the frontal image and a generic profile face model is aligned with the profile image. The deformation procedure results in a single 3-D face model of the specific human face. It precisely reflects the geometric features of the specific subject. After that, subdivision spline surface construction and multi-direction texture mapping techniques are used to smooth the model and endow photometric detail to the specific 3-D geometric face model. This smoothed and texturized specific 3-D face model is then used to generate 2-D images of the subject at a variety of face poses. These synthesized face images can be used to build a set of training images that may be used to train a recognition classifier.

Claims

1. A computer-implemented process for face recognition, comprising using a computer to perform the following process actions:

inputting an image of a face of a subject sought to be recognized having a practicular face pose;

inputting a generic three dimensional face model;

creating a specific three dimensional face model of the specific subject sought to be recognized by deforming the generic face model to conform to the shape of the face depicted in the input image;

synthesizing various face pose images using the specific 3-D face model; and

employing the synthesized images as training images to train a recognizer.

2. The computer-implemented process of claim 1 further comprising the process action of using a spline surface construction technique to smooth the specific face model.

3. The computer-implemented process of claim 2 further comprising the process action of using a texture mapping technique to endow textural detail to the smoothed face model.

4. The computer-implemented process of claim 1 wherein the process action for inputting an image of a face comprises inputting one frontal view and one profile view of the face.

5. The computer-implemented process of claim 1 wherein the process action for inputting an image comprises inputting at least two views of a face each having poses varying in orientation from the others by at least 15 degrees.

6. The computer-implemented process of claim 1 wherein the process action for inputting a generic three dimensional face model comprises employing a polygon model that depicts the surfaces of the face as a series of vertices defining a facial mesh.

7. The computer-implemented process of claim 6 wherein the process action for creating a specific three dimensional face model of the subject comprising using a deformation technique to create a specific three dimensional model of the subject from the generic model and the input images.

8. The computer-implemented process of claim 7 wherein the process action of using a deformation technique to create a specific three dimensional model further comprises the process actions of:

extracting feature point sets from input frontal and profile images of the subject; and

modifying the generic face model to the specific face model by comparison and mapping between the feature point sets.

9. The computer-implemented process of claim 4 wherein symmetry of the face is assumed, such that for the purposes of said creating of a specific 3-D face model it is assumed the side of the face opposite that depicted in the profile image has the same shape and coloring.

10. The computer-implemented process of claim 1 wherein two or more images are used to create the model of a specific face, further comprising a process action of refining the specific model of the face by using additional images.

11. The computer-implemented process of claim 2 wherein the facet-based representation of the specific face is replaced with a spline surface representation.

12. The computer-implemented process of claim 3 wherein the process action to endow textural detail to the face model further comprises the process action of using the input images to assign color intensity to each pixel of the three dimensional face model.

13. The computer-implemented process of claim 12 wherein a Bé

zier patch technique is employed to add texture detail to a face model.

14. The computer-implemented process of claim 13 wherein the process action for assigning color intensity further comprises the process action of:

for each Bé

zier surface patch of the face surface determining a corresponding texture patch by mapping the boundary curve of the Bé

zier patch to the face image.

15. The computer-implemented process of claim 13 wherein both frontal and profile images are employed and wherein the face image chosen to provide the texture information depends on the preferred direction of the Bé

zier patch;

such that the frontal image is used to map when the angle between the direction and the Y-Z plane is less than 30 degrees and wherein a profile image is used when the angle between the direction and the Y-Z plane is greater than 30 degrees.

16. The computer-implemented process of claim 1 wherein face pose images are synthesized for every in-plane rotation of plus or minus 10 to 15 degrees and every out-of-plane rotation of plus or minus 15-20 degrees, with increments of about 3-7 degrees within a group.

17. The computer-implemented process of claim 1 wherein the illumination is varied to produce the synthetic images at varied illuminations by taking the generated images and synthetically varying the illumination to produce each image at various illuminations.

19. The system of claim 18, further comprising a computer module to,

employ the synthesized images as training images by inputting the synthesized images into a recognizer.

21. The system of claim 18, further comprising a computer module to,

use a spline surface technique to smooth the specific face model.

22. The system of claim 21, further comprising a computer module to,

use a texture mapping technique to endow textural detail to the smoothed face model.

18. A system for generating synthesized face images, the system comprising:

at least one camera positioned so capture at least one image of a subject for whom images are to be synthesized;

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input at least one image of a face of a subject from the at least one camera;

input a generic face model;

create a specific face model of the subject by deforming the generic face model to conform to the shape of the face depicted in the input image; and

synthesize various face poses using the specific 3-D face model.

23. A computer-readable medium having computer-executable instructions for generating synthesized images of a face, said computer-executable instructions comprising:

inputting an image of a face of a subject having a practicular face pose;

inputting a generic three dimensional face model;

creating a specific three dimensional face model of the specific subject by deforming the generic face model to conform to the shape of the face depicted in the input image;

using a spline surface construction technique to smooth the specific face model;

using a texture mapping technique to endow textural detail to the smoothed face model;

synthesizing various face pose images using the specific 3-D face model; and

employing the synthesized images as training images to train a recognizer.