Method and system for robust human ethnicity recognition using image feature-based probabilistic graphical models

• US 8,379,937 B1
• Filed: 09/29/2008
• Issued: 02/19/2013
• Est. Priority Date: 09/29/2008
• Status: Active Grant

• Current Assignee: Videomining Corporation
• Original Assignee: Videomining Corporation
• Inventors: Moon, Hankyu, Sharma, Rajeev, Jung, Namsoon
• Primary Examiner(s): Johns, Andrew W
• Assistant Examiner(s): Dunphy, David F
• Application Number:
  US12/286,233
• Time in Patent Office: 1,604 Days
• Field of Search: None
• US Class Current: 382/118
• CPC Class Codes: G06V 10/443 by matching or filtering
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

The present invention is a method and system to provide a face-based automatic ethnicity recognition system that utilizes ethnicity-sensitive image features and probabilistic graphical models to represent ethnic classes. The ethnicity-sensitive image features are derived from groups of image features so that each grouping of the image features contributes to more accurate recognition of the ethnic class. The ethnicity-sensitive image features can be derived from image filters that are matched to different colors, sizes, and shapes of facial features—such as eyes, mouth, or complexion. The ethnicity-sensitive image features serve as observable quantities in the ethnic class-dependent probabilistic graphical models, where each probabilistic graphical model represents one ethnic class. A given input facial image is corrected for pose and lighting, and ethnicity-sensitive image features are extracted. The extracted image features are fed to the ethnicity-dependent probabilistic graphical models to determine the ethnic class of the input facial image.

Claims

1. A method for determining ethnicity of people based on their facial images captured by at least a means for capturing images for a target person, comprising the following steps of:

a) forming image filter groups in a plurality of image filters,

b) computing filter responses by applying the image filters to facial images from an ethnicity-annotated facial images database,

c) computing joint histograms of the filter responses within each filter group,

d) deriving ethnicity-sensitive image features from the image filter groups based on contributions of the groups to a task of recognizing a given ethnic class against other ethnic classes,

e) constructing probabilistic graphical models for recognizing the ethnicity of people using the ethnicity-sensitive image features as observable quantities of the probabilistic graphical models,

f) training the probabilistic graphical models using the ethnicity-annotated facial images so that each of the probabilistic graphical models is dedicated to represent one ethnic class,

g) extracting the ethnicity-sensitive image features of an input face, and

h) determining the ethnic class of the input face based on the outputs computed from the probabilistic graphical models using the ethnicity-sensitive image features,

wherein the ethnicity-sensitive image features are extracted based on specific locations from the facial images, and

wherein the contribution of the grouping is computed based on comparison between the first joint histograms of the image filter responses conditioned on the given ethnic class and the second joint histograms of the image filter responses conditioned on other ethnic classes.

2. The method according to claim 1, wherein the method further comprises a step of constructing filter groups by sampling pairs of filters from image filters matched to facial features so that the image filters from each pair represent different facial features,

wherein each image filter responds to a specific color, size, and shape of one facial feature, and

wherein a subset of the filter pairs is chosen to represent each ethnic class.

3. The method according to claim 1, wherein the method further comprises a step of training the probabilistic graphical models to produce high likelihood scores to facial images belonging to the given ethnic class, and to produce low likelihood scores to facial images belonging to other ethnic classes.

4. The method according to claim 1, wherein the method further comprises a step of using learning machines for estimating and correcting facial pose of the input face,

wherein the corrected input face is fed to an ethnicity-sensitive image feature extraction step.

5. A system for determining ethnicity of people based on their facial images, comprising:

a) an annotation system that comprises a human annotator, an external storage with a facial image database, and a computer system that consists of a visual display, an input device, a control and processing system, and an internal storage,

b) a training system that comprises a computer system having a control and processing system and an internal storage,

wherein the training system is programmed to perform the following steps of;

forming image filter groups in a plurality of image filters,

computing filter responses by applying the image filters to facial images from an ethnicity-annotated facial images database,

computing joint histograms of the filter responses within each filter group,

deriving ethnicity-sensitive image features from the image filter groups based on contributions of the groups to a task of recognizing a given ethnic class against other ethnic classes, constructing probabilistic graphical models for recognizing the ethnicity of people using the ethnicity-sensitive image features as observable quantities of the probabilistic graphical models, and

training the probabilistic graphical models using the ethnicity-annotated facial images so that each of the probabilistic graphical models is dedicated to represent one ethnic class, and

c) an ethnicity classification system that comprises at least a means for capturing images, a computer system having a control and processing system and an internal storage,

wherein the ethnicity classification system is programmed to perform the following steps of;

extracting the ethnicity-sensitive image features of an input face, and

determining the ethnic class of the input face based on the outputs computed from the probabilistic graphical models using the ethnicity-sensitive image features,

wherein the ethnicity-sensitive image features are extracted based on specific locations from the facial images, and

wherein the contribution of the grouping is computed based on comparison between the first joint histograms of the image filter responses conditioned on the given ethnic class and the second joint histograms of the image filter responses conditioned on other ethnic classes.

6. The system according to claim 5, wherein the system further comprises a computer system for constructing filter groups by sampling pairs of filters from image filters matched to facial features so that the image filters from each pair represent different facial features,

wherein each image filter responds to a specific color, size, and shape of one facial feature, and wherein a subset of the filter pairs is chosen to represent each ethnic class.

7. The system according to claim 5, wherein the system further comprises a computer system for training the probabilistic graphical models to produce high likelihood scores to facial images belonging to the given ethnic class, and to produce low likelihood scores to facial images belonging to other ethnic classes.

8. The system according to claim 5, wherein the system further comprises a computer system for using learning machines for estimating and correcting facial pose of the input face,

wherein the corrected input face is fed to an ethnicity-sensitive image feature extraction step.