Method and system for enhancing three dimensional face modeling using demographic classification

• US RE45,768 E1
• Filed: 05/03/2012
• Issued: 10/20/2015
• Est. Priority Date: 04/14/2003
• Status: Active Grant

• Current Assignee: Xenogenic Development Limited Liability Company (Intellectual Ventures LLC)
• Original Assignee: Hyacinth Audio LLC (Intellectual Ventures LLC)
• Inventors: Sharma, Rajeev, Sengupta, Kuntal
• Primary Examiner(s): Vu, Kim
• Assistant Examiner(s): VANCHY JR, MICHAEL J
• Application Number:
  US13/462,983
• Time in Patent Office: 1,265 Days
• Field of Search: 382/118, 382/154
• US Class Current: 1/1
• CPC Class Codes: G06K 9/00288 Classification, e.g. identification
• CPC Class Codes: G06K 9/36 Image preprocessing, i.e. processing the image information without deciding about the identity of the image image data processing or generation, in general G06T
• CPC Class Codes: G06K 9/6206 involving a deformation of the sample or reference pattern; Elastic matching segmentation of touching or overlapping patterns involving deformable models G06K9/34; segmentation involving deformable models for general image processing G06T7/149
• CPC Class Codes: G06K 9/6857 Coarse/fine approaches, e.g. resolution of ambiguities, multiscale approaches
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects

Abstract

The present invention is a system and method for modeling faces from images captured from a single or a plurality of image capturing systems at different times. The method first determines the demographics of the person being imaged. This demographic classification is then used to select an approximate three dimensional face model from a set of models. Using this initial model and properties of camera projection, the model is adjusted leading to a more accurate face model.

Claims

1. A method for face modeling, comprising the steps of:

(a) processing face detection and facial feature detection on a plurality of images for a person with a single or a plurality of image capturing systems,

(b) locating four landmarks on the face of the person based on the facial feature detection,

wherein the face is detected by the face detection, and

wherein three point features from the four landmarks form a basis plane,

(c) training support vector machine (SVM) based demographic classifiers with between one and two thousand images as an input at a learning phase,

(d) processing said plurality of images to obtain demographic recognition of the person in the captured images using the support vector machine (SVM) based demographic classifiers,

(e) choosing a face model specific to the demographic recognition of the person as an approximate face model,

whereby calculation of affine coordinates using demographic dependent constant can be facilitated by the chosen approximate face model, and

(f) combining said demographic recognition with affine coordinate based mesh adjustment technique for said face modeling,

wherein said demographic recognition comprises gender and ethnicity recognition, and

whereby the face modeling is followed by a view generation of the face using rendering tools.

2. The method according to claim 1, wherein the method further comprises a step of using affine lines and their slope adjustment, which is proportional to depth of the point, for model estimation.

3. The method according to claim 1, wherein the method further comprises a step of using the affine line properties for re-projecting a matched pair in two images to a third image, once four facial landmarks are located in all of the three images.

4. The method according to claim 1, wherein the method further comprises a step of using a single view to crudely model the face based on gender and ethnicity and then use anthropometric measures for identification.

5. The method according to claim 1, wherein the method further comprises a step of using multiple views to model the face in the image based on the combination of the demographics and the affine line properties and then use the anthropometric measures for identification purposes.

6. An apparatus for face modeling, comprising:

(a) a single or a plurality of image capturing means directed at a person,

(a)(b) means for processing face detection and facial feature detection on a plurality of images for the person,

(b)(c) means for locating four landmarks on the face of the person based on the facial feature detection,

wherein the face is detected by the face detection, and

wherein three point features from the four landmarks form a basis plane,

(c)(d) means for training support vector machine (SVM) based demographic classifiers with between one and two thousand images as an input at a learning phase,

(d)(e) a processing means for recognizing demographics from at least an image,

(e)(f) a selection means that chooses a face model specific to the demographic recognition of the person as an approximate face model,

whereby calculation of affine coordinates using demographic dependent constant can be facilitated by the chosen approximate face model,

(f)(g) a processing means for combining the demographics recognition with affine coordinate based mesh adjustment technique for said face modeling, and

(g)(h) at least a rendering tool for a view generation of the face,

wherein the demographics recognition comprises gender and ethnicity recognition.

7. The apparatus of claim 6, wherein the apparatus further comprises means for using said affine lines and their slope adjustment, which is proportional to depth of the point, for model estimation.

8. The apparatus of claim 6, wherein the apparatus further comprises means for using the affine line properties for re-projecting a matched pair in two images to a third image, once four facial landmarks are located in all of the three images.

9. The apparatus of claim 6, wherein the apparatus further comprises means for using a single view to crudely model the face in the image based on the gender and ethnicity and then use anthropometric measures for identification.

10. The apparatus of claim 6, wherein the apparatus further comprises means for using multiple views to model the face in the image based on the combination of the demographics and the affine line properties and then use the anthropometric measures for identification purposes.

11. A method for face modeling, comprising:

implementing facial feature detection using one or more images of a person, including locating at least four predetermined landmarks on an image of a face of the one or more images of the person;

training support vector machine (SVM) based demographic classifiers using a plurality of sample images as an input at a learning phase;

obtaining demographic recognition of the person using the one or more images based on the SVM based demographic classifiers, wherein the demographic recognition includes gender and ethnicity recognition;

choosing a face model specific to the demographic recognition of the person to serve as an approximate face model;

calculating an affine coordinate, facilitated by the approximate face model, using a demographic dependent constant;

combining the demographic recognition of the person with an affine coordinate based mesh adjustment technique; and

generating a 3D view of the face of the person using rendering tools.

12. The method of claim 11, wherein the affine coordinate based mesh adjustment technique utilizes affine lines and associated slope adjustment, proportional to depth of a landmark for model estimation.

13. The method of claim 11, wherein the affine coordinate based mesh adjustment technique includes utilizing affine line properties to re-project a matched pair of images to a third image, in response to the at least four predetermined landmarks being located in the image of the face of the one or more images.

14. The method of claim 11,

wherein the obtaining includes utilizing a single view to model the face in the image based on gender and ethnicity, and

wherein the affine coordinate based mesh adjustment technique includes utilizing anthropometric measures.

15. The method of claim 11,

wherein the obtaining includes utilizing a plurality of views to model the face in the image based on the combination of the demographic recognition and the affine coordinate based mesh adjustment technique, and

wherein the affine coordinate based mesh adjustment technique includes utilizing anthropometric measures for identification purposes.

16. The method of claim 11, wherein the at least four predetermined landmarks on the image of the face of the one or more images include a center of both eyes, a nose, and a mouth.

17. The method of claim 16, wherein a basis plane is formed by connecting the center of the both eyes and the mouth.

18. The method of claim 17, wherein the basis plane is utilized by the affine coordinate based mesh adjustment technique.

19. The method of claim 11, wherein the demographic recognition is further obtained by using the SVM based demographic classifiers trained by over a thousand sample images.

20. An apparatus for face modeling, comprising:

a processor; and

a storage unit configured to store executable instructions that, when executed, cause the processor to perform operations including;

implementing facial feature detection on one or more images of a person;

locating at least four designated landmarks on an image of a face of the one or more images based on the facial feature detection;

training support vector machine (SVM) based demographic classifiers using plural sample training images as an input at a learning phase;

recognizing demographics of the person using at least one of the one or more images;

choosing a face model associated with the recognized demographics of the person as an approximate face model;

calculating an affine coordinate, facilitated by the approximate face model, using a demographic dependent constant, changing means to a processor plus storage;

combining the recognized demographics with an affine coordinate based mesh adjustment technique for said face modeling; and

generating a 3D view of the face using rendering tools;

wherein the demographics recognition comprises gender and ethnicity recognition.

21. The apparatus of claim 20, wherein the executable instructions that, when executed, further cause the processor to perform operations including using affine lines and associated slope adjustment, which is proportional to depth of one of the landmarks, for model estimation.

22. The apparatus of claim 20, wherein the executable instructions that, when executed, further cause the processor to perform operations including using affine line properties to re-project a matched pair in two images to a third image, in response to the at least four designated landmarks being located in the image of the face of the one or more images.

23. The apparatus of claim 20, wherein the executable instructions that, when executed, further cause the processor to perform operations including using a single view to model the image of the face of the one or more images based on gender and ethnicity and then use anthropometric measures for identification purposes.

24. The apparatus of claim 20, wherein the executable instructions that, when executed, further cause the processor to perform operations including using a plurality of views to model the image of the face of the one or more images based on the combination of the recognized demographics and the affine coordinate based mesh adjustment technique and then use anthropometric measures for identification purposes.

25. The apparatus of claim 20, wherein the at least four designated landmarks on the image of the face of the one or more images include a center of both eyes, a nose, and a mouth.

26. The apparatus of claim 25, wherein a basis plane is formed by connecting the center of the both eyes and the mouth.

27. The apparatus of claim 26, wherein the basis plane is utilized for the affine coordinate based mesh adjustment technique.

28. The apparatus of claim 20, wherein the SVM based demographic classifiers include a support vector machine trained by over a thousand sample images.