Method and system for robust demographic classification using pose independent model from sequence of face images

• US 7,848,548 B1
• Filed: 06/11/2007
• Issued: 12/07/2010
• Est. Priority Date: 06/11/2007
• Status: Active Grant

• Current Assignee: Videomining Corporation
• Original Assignee: Videomining Corporation
• Inventors: Moon, Hankyu, Sharma, Rajeev, Mummareddy, Satish
• Primary Examiner(s): Mehta, Bhavesh M
• Assistant Examiner(s): Koziol, Stephen R
• Application Number:
  US11/811,614
• Time in Patent Office: 1,275 Days
• Field of Search: 382/100, 382/115, 382/118, 382/181, 382/190, 382/325
• US Class Current: 382/118
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06V 10/75 Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
• CPC Class Codes: G06V 10/7557 based on appearance, e.g. active appearance models [AAM]
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/178 estimating age from face image; using age information for improving recognition

Abstract

The invention provides a face-based automatic demographics classification system that is robust to pose changes of the target faces and to accidental scene variables, by using a pose-independent facial image representation which comprises multiple pose-dependent facial appearance models. Given a sequence of people'"'"'s faces in a scene, the two-dimensional variations are estimated and corrected using a novel machine learning based method. We estimate the three-dimensional pose of the people, using a machine learning based approach. The face tracking module keeps the identity of the person using geometric and appearance cues, where multiple appearance models are built based on the poses of the faces. Each separately built pose-dependent facial appearance model is fed to the demographics classifier, which is trained using only the faces having the corresponding pose. The classification scores from the set of pose-dependent classifiers are aggregated to determine the final face category, such as gender, age, and ethnicity.

Claims

1. A method for automatically performing robust and efficient demographics classification based on a sequence of facial images of people, using a pose-independent facial image representation, comprising the following steps of:

a) capturing a plurality of input images of the people by a plurality of means for capturing images,

b) detecting faces in a plurality of images captured from said means for capturing images

c) estimating a two-dimensional facial pose and three-dimensional facial pose of the detected face by employing a parallel array of multiple learning machine regressors,

d) tracking the faces to keep the identity of the person,

e) constructing multiple pose-dependent appearance models of the faces that belong to a given track, and

f) performing demographics classification based on a pose-independent face representation, wherein the method further comprises multiple pose-dependent facial appearance models built for tracking and classification, based on an estimated pose of the face, and

whereby demographic information can comprise age, gender, and ethnicity information.

2. The method according to claim 1, wherein the method further comprises a step of detecting faces in a plurality of images captured from said means for capturing images, using a machine learning based face detection method.

3. The method according to claim 1, wherein the method further comprises a step of estimating the two-dimensional facial pose of the detected face by employing a parallel array of multiple learning machine regressors,

wherein each machine has an inherent pose parameter, and is trained to output the likelihood of the input face having the inherent pose.

4. The method according to claim 3, wherein the method further comprises a step of correcting the two-dimensional pose of the face detected from face detection, by using the estimated two-dimensional facial pose.

5. The method according to claim 3, wherein the method further comprises a step of generating the training face images by applying a perturbation of the (x, y) position, size, and the orientation to the manually aligned faces,

wherein the distribution of the perturbation is chosen to be the same as the distribution of actual geometric variation of the faces from the face detection step.

6. The method according to claim 3, wherein the method further comprises a step of extracting relevant features having a certain facial pose, for the training and testing, by collecting a sub-regions of the whole facial image,

where the regions are defined by a set of windows, whose locations and sizes match the locations and the sizes of the corresponding facial features having a described global facial pose.

7. The method according to claim 3, wherein the method further comprises a step of extracting relevant features for estimating the facial pose, for the training and testing, by applying the filtering by a set of filters constructed to collect edge responses of a boundary gradient signature of the facial features,

wherein the range of the model pose for generating the filter corresponds to an individual inherent two-dimensional pose of a specific machine, thereby providing a more specialized estimation.

8. The method according to claim 1, wherein the method further comprises a step of estimating the three-dimensional facial pose of the detected face by employing a parallel array of multiple learning machine regressors,

wherein each machine has an inherent three-dimensional pose parameter, and is trained to output the likelihood of the input face having an inherent pose.

9. The method according to claim 8, wherein the method further comprises a step of generating the training face images by applying a two-dimensional facial pose correction to the faces having varied three-dimensional pose, detected from the face detection step.

10. The method according to claim 8, wherein the method further comprises a step of extracting relevant features for estimating the facial pose, for the training and testing, by applying a filtering by a set of filters constructed to collect a boundary gradient signature of the facial features,

wherein the range of the model pose for generating the filter corresponds to an individual inherent three-dimensional pose of the specific machine, thereby providing a more specialized estimation.

11. The method according to claim 1, wherein the method further comprises a step of tracking the faces to keep the identity of the person, by using geometric and appearance match scores.

12. The method according to claim 11, wherein the method further comprises the following steps of:

a) generating a track when a new person appears in the scene,

b) assigning detected faces to existing tracks to keep identities of people in the scene, and

c) terminating a track when a person is out of the scene.

13. The method according to claim 11, wherein the method further comprises a step of finding a highest score match between the track history and the new input face using both geometric and appearance match scores.

14. The method according to claim 1, wherein the method further comprises a step of constructing multiple pose-dependent appearance models of the faces that belong to a given track, based on the estimated three-dimensional pose of the face.

15. The method according to claim 14, wherein the method further comprises a step of sending each new face in the track to one of the pose-dependent appearance models based on the estimated pose, and incrementally add the pixel values to the corresponding appearance model after applying histogram equalization and linear lighting adjustment to reduce the effect of lighting variation.

16. The method according to claim 1, wherein the method further comprises a step of performing demographics classification based on a set of pose-dependent synthetic facial appearance models, which is pose-independent as a whole.

17. The method according to claim 16, wherein the method further comprises a step of performing the classifications across different demographics categories and across different poses,

wherein the classification is performed on the pose-dependent synthetic appearance models built during the tracking, instead of being performed on individual face image instances.

18. The method according to claim 16, wherein the method further comprises a step of aggregating the demographics scores from all the pose-dependent demographics classifications,

wherein each score is weighted by the confidence level of that particular appearance models.

19. The method according to claim 16, wherein the method further comprises a step of training multiple demographics learning machines,

wherein each machine is trained to estimate the likelihood of the given pose-dependent appearance model possessing a certain demographics category.

20. An apparatus for automatically performing robust and efficient demographics classification based on a sequence of the facial images of people, using a pose-independent facial image representation, comprising:

a) means for capturing a plurality of input images of the people by a plurality of means for capturing images,

b) means for detecting faces in a plurality of images captured from said means for capturing images

c) means for estimating a two-dimensional facial pose and three-dimensional facial pose of the detected face by employing a parallel array of multiple learning machine regressors,

d) means for tracking the faces to keep the identity of the person,

e) means for constructing multiple pose-dependent appearance models of the faces that belong to a given track, and

f) means for performing demographics classification based on a pose-independent face representation,

wherein the apparatus further comprises means for building multiple pose-dependent facial appearance models built for tracking and classification, based on an estimated pose of the face, and

whereby demographic information can comprise age, gender, and ethnicity information.

21. The apparatus according to claim 20, wherein the apparatus further comprises means for detecting faces in a plurality of images captured from said means for capturing images, using a machine learning based face detection method.

22. The apparatus according to claim 20, wherein the apparatus further comprises means for estimating the two-dimensional facial pose of the detected face by employing a parallel array of multiple learning machine regressors,

wherein each machine has an inherent pose parameter, and is trained to output the likelihood of the input face having the inherent pose.

23. The apparatus according to claim 22, wherein the apparatus further comprises means for correcting the two-dimensional pose of the face detected from face detection, by using the estimated two-dimensional facial pose.

24. The apparatus according to claim 22, wherein the apparatus further comprises means for generating the training face images by applying a perturbation of the (x, y) position, size, and the orientation to the manually aligned faces,

wherein the distribution of the perturbation is chosen to be the same as the distribution of actual geometric variation of the faces from the face detection step.

25. The apparatus according to claim 22, wherein the apparatus further comprises means for extracting relevant features having a certain facial pose, for the training and testing, by collecting a sub-regions of the whole facial image,

where the regions are defined by a set of windows, whose locations and sizes match the locations and the sizes of the corresponding facial features having a described global facial pose.

26. The apparatus according to claim 22, wherein the apparatus further comprises means for extracting relevant features for estimating the facial pose, for the training and testing, by applying the filtering by a set of filters constructed to collect edge responses of a boundary gradient signature of the facial features,

wherein the range of the model pose for generating the filter corresponds to an individual inherent two-dimensional pose of a specific machine, thereby providing a more specialized estimation.

27. The apparatus according to claim 20, wherein the apparatus further comprises means for estimating the three-dimensional facial pose of the detected face by employing a parallel array of multiple learning machine regressors,

wherein each machine has an inherent three-dimensional pose parameter, and is trained to output the likelihood of the input face having an inherent pose.

28. The apparatus according to claim 27, wherein the apparatus further comprises means for generating the training face images by applying a two-dimensional facial pose correction to the faces having varied three-dimensional pose, detected from the face detection step.

29. The apparatus according to claim 27, wherein the apparatus further comprises means for extracting relevant features for estimating the facial pose, for the training and testing, by applying a filtering by a set of filters constructed to collect a boundary gradient signature of the facial features,

wherein the range of the model pose for generating the filter corresponds to an individual inherent three-dimensional pose of the specific machine, thereby providing a more specialized estimation.

30. The apparatus according to claim 20, wherein the apparatus further comprises means for tracking the faces to keep the identity of the person, by using geometric and appearance match scores.

31. The apparatus according to claim 20, wherein the apparatus further comprises:

a) means for generating a track when a new person appears in the scene,

b) means for assigning detected faces to existing tracks to keep identities of people in the scene, and

c) means for terminating a track when a person is out of the scene.

32. The apparatus according to claim 30, wherein the apparatus further comprises means for finding a highest score match between the track history and the new input face using both geometric and appearance match scores.

33. The apparatus according to claim 20, wherein the apparatus further comprises means for constructing multiple pose-dependent appearance models of the faces that belong to a given track, based on the estimated three-dimensional pose of the face.

34. The apparatus according to claim 33, wherein the apparatus further comprises means for sending each new face in the track to one of the pose-dependent appearance models based on the estimated pose, and incrementally adding the pixel values to the corresponding appearance model after applying histogram equalization and linear lighting adjustment to reduce the effect of lighting variation.

35. The apparatus according to claim 20, wherein the apparatus further comprises means for performing demographics classification based on a set of pose-dependent synthetic facial appearance models, which is pose-independent as a whole.

36. The apparatus according to claim 35, wherein the apparatus further comprises means for performing the classifications across different demographics categories and across different poses,

wherein the classification is performed on the pose-dependent synthetic appearance models built during the tracking, instead of being performed on individual face image instances.

37. The apparatus according to claim 35, wherein the apparatus further comprises means for aggregating the demographics scores from all the pose-dependent demographics classifications,

wherein each score is weighted by the confidence level of that particular appearance models.

38. The apparatus according to claim 35, wherein the apparatus further comprises means for training multiple demographics learning machines,

wherein each machine is trained to estimate the likelihood of the given pose-dependent appearance model possessing a certain demographics category.