Pose-invariant face recognition system and process

• US 7,142,697 B2
• Filed: 11/05/2004
• Issued: 11/28/2006
• Est. Priority Date: 09/13/1999
• Status: Expired due to Term

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Hong-Jiang, Huang, Fu Jie, Chen, Tsuhan
• Primary Examiner(s): Wu, Jingge
• Assistant Examiner(s): CARTER, AARON W
• Application Number:
  US10/982,743
• Publication Number:
  US 20050147291A1
• Time in Patent Office: 753 Days
• Field of Search: 382/118, 382/155, 382/156, 382/159, 382/170, 382/181, 382/203, 340/5.52, 340/5.53, 902/3, 713/186
• US Class Current: 382/118
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A face recognition system and process for identifying a person depicted in an input image and their face pose. This system and process entails locating and extracting face regions belonging to known people from a set of model images, and determining the face pose for each of the face regions extracted. All the extracted face regions are preprocessed by normalizing, cropping, categorizing and finally abstracting them. More specifically, the images are normalized and cropped to show only a persons face, categorized according to the face pose of the depicted person'"'"'s face by assigning them to one of a series of face pose ranges, and abstracted preferably via an eigenface approach. The preprocessed face images are preferably used to train a neural network ensemble having a first stage made up of a bank of face recognition neural networks each of which is dedicated to a particular pose range, and a second stage constituting a single fusing neural network that is used to combine the outputs from each of the first stage neural networks. Once trained, the input of a face region which has been extracted from an input image and preprocessed (i.e., normalized, cropped and abstracted) will cause just one of the output units of the fusing portion of the neural network ensemble to become active. The active output unit indicates either the identify of the person whose face was extracted from the input image and the associated face pose, or that the identity of the person is unknown to the system.

Claims

1. A computer-implemented face recognition process for identifying a person depicted in an input image, comprising:

creating means for creating a database of a plurality of model image characterizations, each of which represents the face of a known person that it is desired to identify in the input image as well as the person'"'"'s face pose;

training means for training a neural network ensemble to identify a person and their face pose from a region which has been extracted from said input image and characterized in a manner similar to the plurality of model images, wherein the network ensemble comprises, a first stage having a plurality of classifiers each of which has input and output units and is dedicated to a particular pose range and outputs a measure of the similarity indicative of the similarity between said characterized input image region and each of said model image characterizations associated with the particular pose range of the classifier, and a fusing neural network as its second stage which combines the outputs of the classifiers to generate an output indicative of the person associated with the characterized input image region and the face pose of that person and which has at least enough output units to allow a different output to represent each person it is desired to identify at each of the pose ranges, and wherein training the neural network ensemble comprises,

preparing each model image characterization from a model image depicting the face of a known person that it is desired to identify in the input image by,

extracting the portion of the model image depicting said face,

normalizing the extracted portion of the model image by resizing it to a prescribed scale if not already at the prescribed scale and adjusting the region so that the eye locations of the depicted subject fall within a prescribed area, and

cropping the extracted portion of the model image by eliminating unneeded portions of the image not specifically depicting part of the face of the subject to create a model face image,

categorizing the model face images by assigning each to one of a set of pose ranges into which its associated face pose falls, and

for each pose range,

choosing a prescribed number of the model face images of each person being modeled which have been assigned to the selected pose range,

concatenating each of the chosen model face images to create a respective dimensional column vector (DCV) for each,

computing a covariance matrix from the DCVs,

calculating eigenvectors and corresponding eigenvalues from the covariance matrix,

ranking the eigenvalues in descending order,

identifying a prescribed number of the top eigenvalues,

using the eigenvectors corresponding to the identified eigenvalues to form the rows of a basis vector matrix (BVM) for the pose range, and

multiplying each DCV by each BVM to Produce a set of principal components analysis (PCA) coefficient vectors for each model face image, and

for each face recognition neural network, inputting, one at a time, each of the PCA coefficient vectors associated with the pose range of the face recognition neural network into the inputs of the network until the outputs of the network stabilize,

initializing the fusing neural network for training,

for each DCV, simultaneously inputting the PCA coefficient vectors generated from the DCV into the respective face recognition neural network associated the vector'"'"'s particular pose range group until all the PCA coefficient vectors of every DCV have been input, and repeating until the outputs of the fusing neural network stabilize, and

for each DCV, simultaneously inputting the PCA coefficient vectors generated from the DCV into the respective face recognition neural network associated the vector'"'"'s particular pose range group and assigning the active output of the fusing neural network as corresponding to the particular person and pose associated with the model image used to create the set of PCA coefficient vectors; and

identifying means for employing the network ensemble to identify the person associated with the characterized input image region and the face pose of that person.

2. The process of claim 1, wherein the identifying means comprises:

preparing means for preparing the face region extracted from an input image by normalizing and cropping the extracted regions, wherein said normalizing comprises resizing the extracted face region to the same prescribed scale if not already at the prescribed scale and adjusting the region so that the eye locations of the depicted subject fall within a prescribed area, and wherein the cropping comprises eliminated unneeded portions of the image not specifically depicting part of the face of the subject;

concatenating means for concatenating the prepared face region to create a DCV;

multiplying means for multiplying the DCV by each BVM to produce a set of PCA coefficient vectors for the extracted face region;

inputting means for inputting each PCA coefficient vector in the set of PCA coefficient vectors into the respective face recognition neural network associated that vector'"'"'s particular pose range group; and

identifying means for identifying the active unit of the output of the fusing neural network and designating the person and pose previously assigned to that unit as the person and pose associated with the extracted face region.

3. A computer-implemented face recognition process for identifying a person depicted in an input image:

creating means for creating a database of a plurality of model image characterizations, each of which represents the face of a known person that it is desired to identify in the input image as well as the person'"'"'s face pose;

training means for training a neural network ensemble to identify a person and their face pose from a region which has been extracted from said input image and characterized in a manner similar to the plurality of model images, wherein the network ensemble comprises, a first stage having a plurality of classifiers each of which has input and output units and is dedicated to a particular pose range and outputs a measure of the similarity indicative of the similarity between said characterized input image region and each of said model image characterizations associated with the particular pose range of the classifier, and a fusing neural network as its second stage which combines the outputs of the classifiers to generate an output indicative of the person associated with the characterized input image region and the face pose of that person, and wherein training the network ensemble comprises,

deriving each model image characterization from a set of model images of people where each model image of the same person shows that person at a different face pose, said deriving comprising,

extracting the portion of each model image depicting a face,

normalizing the extracted portion of each model image by resizing it to a prescribed scale if not already at the prescribed scale and adjusting the region so that the eye locations of the depicted subject fall within a prescribed area,

cropping the extracted portion of each model image by eliminating unneeded portions of the image not specifically depicting part of the face of the subject to create a model face image,

concatenating each of the model face images to create a respective model dimensional column vector (DCV) for each,

categorizing the model DCVs by assigning each to one of a set of pose ranges into which its associated face pose falls, and

inputting the model DCV of the each model face image falling in a particular pose range, one at a time, to a pre-selected classifier dedicated to the particular pose range,

initializing the fusing neural network for training,

simultaneously inputting the respective DCV of each model face image into all classifiers, until the DCV of every model image has been input, and repeating until the outputs of the neural network stabilize, and

simultaneously inputting the respective DCV of each model face image into all classifiers, and assigning the active output the neural network as corresponding to the particular person and pose associated with the model image used to create the DCV; and

identifying means for employing the network ensemble to identify the person associated with the characterized input image region and the face pose of that person.

4. The process of claim 3, wherein the identifying means comprises:

preparing means for preparing the face region extracted from an input image by normalizing and cropping the extracted regions, wherein said normalizing comprises resizing the extracted face region to the same prescribed scale if not already at the prescribed scale and adjusting the region so that the eye locations of the depicted subject fall within a prescribed area, and wherein the cropping comprises eliminated unneeded portions of the image not specifically depicting part of the face of the subject;

concatenating means for concatenating the prepared face region to create a DCV;

inputting means for inputting the DCV of the face region into all classifiers; and

identifying means for identifying the active output of the neural network and designating the person previously assigned to that unit as the person associated with the extracted face region.

5. The process of claim 4, further comprising specifying means for specifying that the person designated as associated with the extracted face region has the face pose previously assigned to the identified active output.