Method and system for customizing facial feature tracking using precise landmark finding on a neutral face image

• US 6,714,661 B2
• Filed: 07/24/2001
• Issued: 03/30/2004
• Est. Priority Date: 11/06/1998
• Status: Expired due to Term

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Nevengineering Incorporated
• Inventors: Neven, Hartmut, Buddenmeier, Ulrich F.
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): Choobin, Barry
• Application Number:
  US09/915,205
• Publication Number:
  US 20020015512A1
• Time in Patent Office: 980 Days
• Field of Search: 382/103, 382/100, 382/240, 382/232, 382/118, 382/209, 382/199, 348/169, 345/473
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/20064 Wavelet transform [DWT]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

The present invention is embodied in a method and system for customizing a visual sensor for facial feature tracking using a neutral face image of an actor. The method may include generating a corrector graph to improve the sensor'"'"'s performance in tracking an actor'"'"'s facial features.

Claims

1. A method for customizing facial feature tracking, comprising:

capturing a front neutral face image of an actor;

automatically finding facial feature locations on the front neutral face image using elastic bunch graph matching;

automatically positioning nodes at the facial feature locations on the front neutral face image of the actor; and

manually correcting the positioning of the nodes on front neutral face image of the actor.

2. A method for customizing facial feature tracking as defined in claim 1, further comprising generating a corrector graph based on the corrected node positions.

9. A method for customizing facial feature tracking as defined in claim 1, wherein manually correcting the positioning of the nodes is performed after the automatically positioning of the nodes at the facial features.

10. A method for customizing facial feature tracking as defined in claim 1, wherein manually correcting the positioning of the nodes is performed using a visual sensor customization wizard presenting the captured front neutral face image of the actor showing the positioning of the nodes at the facial feature locations on the front neutral face image of the actor, and presenting an example image to indicate proper node positioning.

11. A method for customizing facial feature tracking as defined in claim 2, wherein the corrector graph associates jets for facial features from a front neutral face image with respective jets for facial features from expressive face images.

12. A method for customizing facial feature tracking as defined in claim 2, further comprising tracking facial features of the actor using the corrector graph.

3. A system for customizing facial feature tracking, comprising:

means for capturing a front neutral face image of an actor;

means for automatically finding facial feature locations on the front neutral face image using elastic bunch graph matching;

means for automatically positioning nodes at the facial feature locations on the front neutral face image of the actor; and

means for manually correcting the positioning of the nodes on front neutral face image of the actor.

4. A system for customizing facial feature tracking as defined in claim 3, further comprising means for generating a corrector graph based on the corrected node positions.

13. A system for customizing facial feature tracking as defined in claim 3, wherein the means for manually correcting the positioning of the nodes includes a visual sensor customization wizard for presenting the captured front neutral face image of the actor showing the positioning of the nodes at the facial feature locations on the front neutral face image of the actor, and presenting an example image to indicate proper node positioning.

14. A system for customizing facial feature tracking as defined in claim 4, wherein the corrector graph associates jets for facial features from a front neutral face image with respective jets for facial features from expressive face images.

15. A system for customizing facial feature tracking as defined in claim 4, further comprising means for tracking facial features of the actor using the corrector graph.

5. A method for customizing facial feature tracking, comprising:

capturing a front neutral face image of an actor;

automatically finding facial feature locations on the front neutral face image using image analysis based on wavelet component values generated from wavelet transformations of the front neutral face image;

automatically positioning nodes at the facial feature locations on the front neutral face image of the actor; and

manually correcting the positioning of the nodes on front neutral face image of the actor.

6. A method for customizing facial feature tracking as defined in claim 5, wherein the wavelet transformations use Gabor wavelets.

16. A method for customizing facial feature tracking as defined in claim 5, wherein manually correcting the positioning of the nodes is performed after the automatically positioning of the nodes at the facial features.

17. A method for customizing facial feature tracking as defined in claim 5, wherein manually correcting the positioning of the nodes is performed using a visual sensor customization wizard presenting the captured front neutral face image of the actor showing the positioning of the nodes at the facial feature locations on the front neutral face image of the actor, and presenting an example image to indicate proper node positioning.

18. A method for customizing facial feature tracking as defined in claim 5, further comprising generating a corrector graph based on the corrected node positions.

19. A method for customizing facial feature tracking as defined in claim 18, wherein the corrector graph associates jets for facial features from a front neutral face image with respective jets for facial features from expressive face images.

20. A method for customizing facial feature tracking as defined in claim 18, further comprising tracking facial features of the actor using the corrector graph.

7. A method for customizing facial feature tracking, comprising:

capturing a front neutral face image of an actor;

finding facial feature locations on the front neutral face image using image analysis based on wavelet component values generated from wavelet transformations of the front neutral face image; and

generating a corrector graph for expressive facial features based on the wave component values at the facial feature locations on the front neutral face image.

8. A method for customizing facial feature tracking as defined in claim 7, wherein the wavelet transformations use Gabor wavelets.

21. A system for customizing facial feature tracking as defined claim 7, wherein the means for manually correcting the positioning of the nodes includes a visual sensor customization wizard for presenting the captured front neutral face image of the actor showing the positioning of the nodes at the facial feature locations on the front neutral face image of the actor, and presenting an example image to indicate proper node positioning.

22. A system for customizing facial feature tracking as defined in claim 7, further comprising means for generating a corrector graph based on the corrected node positions.

23. A system for customizing facial feature tracking as defined in claim 22, wherein the corrector graph associates jets for facial features from a front neutral face image with respective jets for facial features from expressive face images.

24. A system for customizing facial feature tracking as defined in claim 22, further comprising means for tracking facial features of the actor using the corrector graph.