Stereo-coupled face shape registration

• US 7,218,760 B2
• Filed: 06/30/2003
• Issued: 05/15/2007
• Est. Priority Date: 06/30/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Hong-Jiang, Ziqing, Li, Gu, Lie
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US10/610,494
• Publication Number:
  US 20040264745A1
• Time in Patent Office: 1,415 Days
• Field of Search: 382/118, 382/151, 382/154, 382/159, 382/190, 382/199, 382/209, 382/218, 382/294, 382/309, 345/419
• US Class Current: 382/118
• CPC Class Codes: G06V 10/7553 based on shape, e.g. active shape models [ASM]
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

A face model having outer and inner facial features is matched to that of first and second models. Each facial feature of the first and second models is represented by plurality of points that are adjusted for each matching outer and inner facial feature of the first and second models using 1) the corresponding epipolar constraint for the inner features of the first and second models. 2) Local grey-level structure of both outer and inner features of the first and second models. The matching and the adjusting are repeated, for each of the first and second models, until the points for each of the outer and inner facial features on the respective first and second models that are found to match that of the face model have a relative offset there between of not greater than a predetermined convergence tolerance. The inner facial features can include a pair of eyes, a nose and a mouth. The outer facial features can include a pair of eyebrows and a silhouette of the jaw, chin, and cheeks.

Claims

1. A method for face modeling and identification of facial features, comprising:

determining outer and inner facial features of a face model;

initializing outer and inner facial features of the face model by taking images of a face to correspond to;

that of a first model for a first face image of the face in a frontal position; and

that of a second model for a second face image of the face moving in a yaw direction, wherein the first and the second face images are pictures taken at substantially the same respective camera orientation;

matching outer and inner facial features by a correlation between the first face image and the second face image;

adjusting the matching outer and inner facial features of the first and the second models using the corresponding epipolar constraint for the first and the second models for a more accurate correlation of facial features between the first face image and the second face image; and

identifying facial features for face recognition.

2. The method as defined in claim 1, wherein:

the inner facial features comprise a nose, a mouth and a pair of eyes; and

the outer facial features comprise an arcuate bottom edge of the face and a pair of eyebrows.

3. The method as defined in claim 1, wherein a plurality points correspond to each said inner facial feature and comprise:

a plurality of mouth corners corresponding to the mouth;

a plurality of eyes corners corresponding to the eye; and

a nose tip corresponding to the nose.

4. The method as defined in claim 1, wherein the first and second face images are the same face captured at the substantially the same respective camera orientation.

5. The method as defined in claim 4, wherein the respective face poses of the first and second face images are:

substantially the same as to pitch;

substantially the same as to roll; and

different as to yaw in a range from about 3 degrees to about 10 degrees.

6. The method as defined in claim 1, wherein the first and second face images are pictures taken at the substantially the same respective camera orientation of the same face in substantially the same face pose range but differing by the yaw.

7. The method as defined in claim 1, wherein the face pose range differs in the first and second face images by a yaw difference of not more than about ten degrees (10°

).

8. The method as defined in claim 1, wherein:

the first face image is a full frontal picture of a human face; and

the second face image differs from the first face image by a yaw of not more than about ten degrees (10°

).

9. The method as defined in claim 1, further comprising outputting a representation of each of the first and second models, each including the respective inner and outer facial features.

10. The method as defined in claim 9, wherein the outputting further comprises:

repeating the matching and the adjusting until each match of the outer and inner facial features on;

the first model is not greater than a predetermined convergence toleration with respect to that of the face model; and

the second model is not greater than a predetermined convergence toleration with respect to that of the face model.

11. The method as defined in claim 1, further comprising repeating the matching and the adjusting until each said match of the outer and inner facial features on:

the first model is not greater than a predetermined convergence toleration with respect to that of the face model; and

the second model is not greater than a predetermined convergence toleration with respect to that of the face model.

12. The method as defined in claim 1, further comprising identifying the face of a person in a database that matches at least one of the first and second face images by using the respective outer and inner facial features of the first and second models.

13. A computer readable medium comprising instructions that, when executed, perform the method of claim 1.

14. A method comprising:

determining outer and inner facial features of a face model, each being represented by plurality of points;

initializing outer and inner facial features of the face model by taking images of a face to correspond to;

that of a first model for a first face image of the face in a frontal position; and

that of a second model for a second face image of the same face moving in a yaw direction;

matching outer and inner facial features by a correlation between the first face image and the second face image;

adjusting the plurality of points for each matching outer and inner facial features of the first and second models using the corresponding epipolar constraint for the first and second models for a more accurate correlation of facial features between the first face image and the second face image;

repeating the matching and the adjusting until;

a majority of the points for each of the outer and inner facial features on the first model that matches that of the face model has a relative offset there between of not greater than a predetermined convergence toleration; and

a majority of the points for each of the outer and inner facial features on the second model that matches that of the face model has a relative offset there between not greater than the predetermined convergence toleration;

outputting the first and the second model that has a relative offset there; and

identifying facial features for face recognition.

15. The method as defined in claim 14, further comprising outputting a representation of each of the first and second models, each including the respective inner and outer facial features.

16. The method as defined in claim 14, wherein:

the inner facial features comprise a nose, a mouth and a pair of eyes; and

the outer facial features comprise an arcuate bottom edge of the face and a pair of eyebrows.

17. The method as defined in claim 14, wherein the plurality points and corresponding inner facial features are selected from the group consisting of:

a plurality of mouth corners corresponding to a mouth;

a plurality of eyes corners corresponding to an eye; and

a nose tip corresponding to a nose.

18. The method as defined in claim 14, wherein the first and second face images are the same face captured at the substantially the same respective camera orientation.

19. The method as defined in claim 18, wherein the respective face poses of the first and second face images are:

substantially the same as to pitch;

substantially the same as to roll; and

different as to yaw in a range from about 3 degrees to about 10 degrees.

20. The method as defined in claim 14, wherein the first and second face images are pictures taken at the substantially the same respective camera orientation of the same face in substantially the same face pose range but differing by the yaw.

21. The method as defined in claim 14, wherein the face pose range differs in the first and second face images by a yaw difference of not more than about ten degrees (10°

).

22. The method as defined in claim 14, wherein:

the first face image is a full frontal picture of a human face; and

the second face image differs from the first face image by a yaw of not more than about ten degrees (10°

).

23. The method as defined in claim 14, further comprising identifying the face of a person in a database that matches at least one of the first and second face images by using the points that respectively correspond to the outer and inner facial features of the first and second models.

24. A computer readable medium comprising instructions that, when executed, perform the method of claim 14.

25. A method comprising the steps of:

determining outer and inner facial features of a face model;

initializing, using a face model having outer and inner facial features wherein each of the outer and inner facial features includes a plurality of points, first and second models for respective first and second face images, wherein a first model is for a first image of a face in a frontal position and a second model is for a second image of the face moving in a yaw direction;

estimating a fundamental matrix for the first and second models;

updating the plurality of points for each inner facial features of the first and second models by using the respective local texture for each said point;

updating each inner facial features of the first and second models;

updating, using the fundamental matrix and the corresponding epipolar constraint, the plurality of points for each of the inner facial features of the first and second models; and

updating each said outer facial features of the first and second models

wherein the updating is for a more accurate correlation of facial features between the first face image and the second face image; and

identifying facial features for facial recognition.

26. The method as defined in claim 25, wherein the steps further comprise:

if the first model has converged within a predetermined tolerance of the face model, outputting a representation of the inner and outer facial features of the first model, otherwise repeating the four updating steps for the first model; and

if the second model has converged within a predetermined tolerance of the face model, outputting a representation of the inner and outer facial features of the second model, otherwise repeating the four updating steps for the second model.

27. The method as defined in claim 25, wherein:

the inner facial features comprise a nose, a mouth and a pair of eyes; and

the outer facial features comprise an arcuate bottom edge of the face and a pair of eyebrows.

28. The method as defined in claim 27, wherein the plurality of points corresponding to the inner facial features of:

the mouth comprises a plurality of mouth corners;

each eye comprises a plurality of eyes corners; and

the nose comprises a nose tip.

29. The method as defined in claim 25, wherein the first and second face images are the same face captured at the substantially the same respective camera orientation.

30. The method as defined in claim 29, wherein the respective face poses of the first and second face images are:

substantially the same as to pitch;

substantially the same as to roll; and

different as to yaw in a range from about 3 degrees to about 10 degrees.

31. The method as defined in claim 25, wherein the first and second face images are pictures taken at the substantially the same respective camera orientation of the same face in substantially the same face pose range but differing by the yaw.

32. The method as defined in claim 25, wherein the face pose range differs in the first and second face images by a yaw difference of not more than about ten degrees (10°

).

33. The method as defined in claim 25, wherein:

the first face image is a full frontal picture of a human face; and

the second face image differs from the first face image by a yaw of not more than about ten degrees (10°

).

34. The method as defined in claim 25, wherein the local texture for each said point comprises photometric data at or proximal to the point.

35. The method as defined in claim 25, further comprising identifying the face of a person in a database that matches at least one of the first and second face images by using, respectively, the output representation of the inner and outer facial features of:

the first model; and

the second model.

36. A computer readable medium comprising instructions that, when executed, perform the method of claim 25.

37. An apparatus comprising:

memory including;

one or more programs;

outer and inner facial features of a face model, each being represented by a plurality of points;

a first face image of a face in a frontal position; and

a second face image of the face moving in a yaw direction;

one or more processors configured to execute one or more programs in the memory to perform actions comprising;

match the plurality of points of the outer and inner facial features of the face model with;

that of a first model for the first face image of the face in the frontal position; and

that of a second model for the second face image of the face moving in a yaw direction;

adjust the plurality of points for each matching outer and inner facial features of the first and the second models using the corresponding epipolar constraint for the first and the second models for a more accurate correlation of facial features between the first face image and the second face image;

repeat the match and adjust steps until;

a majority of the points for each of the outer and inner facial features on the first model that matches that of the face model has a relative offset there between of not greater than a predetermined convergence toleration; and

a majority of the points for each of the outer and inner facial features on the second model that matches that of the face model has a relative offset there between not greater than the predetermined convergence toleration; and

a monitor to display the facial features for face recognition.

38. The apparatus as defined in claim 37, wherein the steps further comprise outputting a representation of each of the fast and second models, each including the respective inner and outer facial features.

39. The apparatus as defined in claim 37, wherein:

the inner facial features comprise a nose, a mouth and a pair of eyes; and

the outer facial features comprise an arcuate bottom edge of the face and a pair of eyebrows.

40. The apparatus as defined in claim 37, wherein the plurality points and corresponding said inner facial feature are selected from the group consisting of:

a plurality of mouth corners corresponding to a mouth;

a plurality of eyes corners corresponding to an eye; and

a nose tip corresponding to a nose.

41. The apparatus as defined in claim 37, wherein the first and second face images are the same face captured at the substantially the same respective camera orientation.

42. The apparatus as defined in claim 41, wherein the respective face poses of the first and second face images are:

substantially the same as to pitch;

substantially the same as to roll; and

different as to yaw in a range from about 3 degrees to about 10 degrees.

43. The apparatus as defined in claim 37, wherein the first and second face images are pictures taken at the substantially the same respective camera orientation of the same face in substantially the same face pose range but differing by the yaw.

44. The apparatus as defined in claim 37, wherein the face pose range differs in the first and second face images by a yaw difference of not more than about ten degrees (10°

).

45. The apparatus as defined in claim 37, wherein:

the first face image is a frill frontal picture of a human face; and

the second face image differs from the first face image by a yaw of not more than about ten degrees (10°

).

46. An apparatus comprising:

means for initializing, using a face model having outer and inner facial features each including a plurality of points, first and second models for respective first and second face images, wherein a first model is for a first face image of a face in a frontal position and a second model is for a second face image of the face moving in a yaw direction;

means for generating the first face image and the second face image with a camera;

means for estimating a fundamental matrix for the first and second models;

means for updating the plurality of points for each said inner and outer facial feature of the first and second models by using the respective local texture for each said point;

means for updating each said inner facial feature of the first and second models;

means for updating, using the fundamental matrix and the corresponding epipolar constraint, the plurality of points for each of the inner facial features of the first and second models; and

means for updating, using the plurality of points for each of the inner facial features of the first and second models, each said outer facial feature of the first and second models,

wherein the updating is for a more accurate correlation of facial features between the first face image and the second face image; and

means for displaying on a monitor used for facial recognition.

47. The apparatus as defined in claim 46, further comprising:

means, if a majority of the points for each of the outer and inner facial features on the first model that matches that of the face model has a relative offset there between of not greater than a predetermined convergence toleration, for outputting a representation of the inner and outer facial features of the first model; and

means, if a majority of the points for each of the outer and inner facial features on the first model that matches that of the face model has a relative offset there between not greater than the predetermined convergence toleration, for outputting a representation of the inner and outer facial features of the second model.

48. The apparatus as defined in claim 46, wherein:

the inner facial features comprise a nose, a mouth and a pair of eyes; and

the outer facial features comprise an arcuate bottom edge of the face and a pair of eyebrows.

49. The apparatus as defined in claim 48, wherein the plurality points corresponding to the inner facial feature of:

the mouth comprises a plurality of mouth corners;

each said eye comprises a plurality of eyes corners; and

the nose comprises a nose tip.

50. The apparatus as defined in claim 46, wherein the first and second face images are the same face captured at the substantially the same respective camera orientation.

51. The apparatus as defined in claim 46, wherein the respective face poses of the first and second face images are:

substantially the same as to pitch;

substantially the same as to roll; and

different as to yaw in a range from about 3 degrees to about 10 degrees.

52. The apparatus as defined in claim 46, wherein the first and second face images are pictures taken at the substantially the same respective camera orientation of the same face in substantially the same face pose range but differing by the yaw.

53. The apparatus as defined in claim 46, wherein the face pose range differs in the first and second face images by a yaw different of not more than about ten degrees (10°

).

54. The apparatus as defined in claim 46, wherein:

the first face image is a full frontal picture of a human face; and

the second face image differs from the first face image by a yaw of not more than about ten degrees (10°

).

55. The apparatus as defined in claim 46, wherein the local texture for each said point comprises photometric data at or proximal to the point.