Method for tracking motion of a face

• US 7,127,081 B1
• Filed: 10/12/2000
• Issued: 10/24/2006
• Est. Priority Date: 10/12/2000
• Status: Expired due to Fees

• Current Assignee: Momentum Bilgisayar Yazilim Danismanlik Ticaret As
• Original Assignee: Momentum Bilgisayar Yazilim Danismanlik Ticaret As
• Inventors: Erdem, A. Tanju
• Primary Examiner(s): Patel, Kanjibhai
• Assistant Examiner(s): AZARIAN, SEYED H
• Application Number:
  US09/689,565
• Time in Patent Office: 2,203 Days
• Field of Search: 382/103, 382/100, 382/107, 382/115, 382/116, 382/117, 382/118, 382/209, 382/203, 382/276, 382/291, 382/106, 382/154, 382/162, 382/168, 382/189, 382/236, 382/243, 382/255, 382/274, 382/285, 382/287, 382/295, 382/305, 345/622, 345/630, 345/156, 345/419, 345/420, 345/474, 340/945, 340/326, 340/946, 340/981, 340/309.7, 340/14.07, 600/424, 348/94, 348/169
• US Class Current: 382/103
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

A method for tracking the motion of a person'"'"'s face for the purpose of animating a 3-D face model of the same or another person is disclosed. The 3-D face model carries both the geometry (shape) and the texture (color) characteristics of the person'"'"'s face. The shape of the face model is represented via a 3-D triangular mesh (geometry mesh), while the texture of the face model is represented via a 2-D composite image (texture image). Both the global motion and the local motion of the person'"'"'s face are tracked. Global motion of the face involves the rotation and the translation of the face in 3-D. Local motion of the face involves the 3-D motion of the lips, eyebrows, etc., caused by speech and facial expressions. The 2-D positions of salient features of the person'"'"'s face and/or markers placed on the person'"'"'s face are automatically tracked in a time-sequence of 2-D images of the face. Global and local motion of the face are separately calculated using the tracked 2-D positions of the salient features or markers. Global motion is represented in a 2-D image by rotation and position vectors while local motion is represented by an action vector that specifies the amount of facial actions such as smiling-mouth, raised-eyebrows, etc.

Claims

1. A method for tracking motion of a face comprising the steps of:

determining the calibration parameter of a camera;

marking salient features of an object with markers for motion tracking;

acquiring a plurality of initial 2-D images of the object;

calculating 3-D locations of the salient features of the object in accordance with the calibration parameter of the camera;

calculating 3-D locations of the global and local markers in the neutral state; and

calculating 3-D locations of the local markers in each action state;

receiving a chronologically ordered sequence of 2-D images of the object;

storing or transmitting tracked motion of the object;

calculating the 3-D locations of the local markers in each action state by estimating the orientation and position of the face in each 2-D image of the action state to conform to the 3-D and 2-D locations of the global markers under a perspective projection model and calculating the 3-D locations of the local markers to conform to the estimated orientation and position of the face and the 2-D locations of the local markers under a perspective projection model.

2. The method of claim 1 further comprising the steps of:

locking onto the markers, and

detecting loss of lock and hence the need for re-locking onto the markers.

3. The method of claim 2 comprising the further step of repeating the locking and tracking steps after the detecting step.

4. The method of claim 1 wherein the step of tracking comprises the steps of:

determining a surface normal for each salient features;

tracking motion of the markers on the object in each 2-D image; and

tracking the 3-D global motion of the markers on the object in each image; and

tracking the 3-D local motion of the markers on the object in each image.

5. The method of claim 4 wherein the step of tracking the 3-D global motion comprises the steps of:

predicting the location of global salient features in a 2-D image;

detecting global salient features in the 2-D image; and

estimating the 3-D global motion of the face in the 2-D image.

6. The method of claim 5 wherein the step of predicting comprises calculating 2-D locations of the global salient features under a perspective projection model using the position and orientation of the face in a previous 2-D image, and the step of detecting comprise detecting the global markers.

7. The method of claim 6 wherein detecting the global markers comprises:

determining visibility indices of global markers;

designing correlation filters for the global markers;

detecting the global markers by applying elliptical correlation filters in a neighborhood of the global markers; and

eliminating superfluous and multiple detected locations.

8. The method of claim 6 wherein the step of estimating the 3D global motion comprises calculating the position and orientation of the face to conform to the 3D locations and the detected locations of the global markers under a perspective projection model.

9. The method of claim 4 wherein the step of tracking the 3-D local motion comprises the steps of:

predicting the location of local salient features;

detecting local salient features; and

estimating the 3-D local motion of the face.

10. The method of claim 9 wherein the local markers are placed on eyebrows and lips.

11. The method of claim 10 wherein the locations of the local markers comprise proximate ends of the eyebrows, corners of the lips, and the upper and lower centers of each lip.

12. The method of claim 11 wherein the step of predicting the locations of local markers comprises calculating the locations of the local markers using the position, orientation, and action states of the face in a previous 2-D image and the step of detecting comprise detecting the local markers.

13. The method of claim 12 wherein detecting the local markers comprise:

determining visibility indices of local markers;

designing correlation filters for the local markers;

detecting the local makers by applying elliptical correlation filters in a neighborhood of the local markers; and

eliminating superfluous and multiple detected locations.

14. The method of claim 13 wherein the step of estimating comprises:

finding 3-D locations of local markers to conform to the detected 2-D locations of the local markers;

calculating an action vector representing the weights of the facial actions in the 2-D image conforming to the found 3-D locations of the local markers and the 3-D locations of the local markers for the neutral and the action states under a perspective projection model.

15. The method of claim 14 wherein the step of calculating an action vector comprises the steps of:

calculating the difference between the 2-D locations of the local markers detected in an image and the 2-D locations of the same markers corresponding to the neutral face;

modifying the difference to conform to the orthographic projection;

calculating the 3-D displacements of the local markers with respect to their location in the neutral face; and

calculating the amount of facial actions conforming to the 3-D displacements of the local markers.

16. The method of claim 14 wherein the step of calculating an action vector comprises the steps of:

calculating the 2-D locations of the local markers corresponding to the neutral face using the global motion found for the current image;

calculating the 2-D locations of the local markers corresponding to the action faces using the global motion found for the current image;

calculating the distance between the detected locations, the distance between the neutral locations, and the distance between the action locations of the markers at the right and left corners of the lips;

calculating the distance between the detected locations, the distance between the neutral locations, and the distance between the action locations of the markers at the upper and lower center of lips;

calculating the distance between the detected locations, the distance between the neutral locations, and the distance between the action locations of the markers at the proximate ends of the eyebrows;

determining the fractional displacements of the local markers for the lips area and for the eyebrows area; and

determining action mode and amount for the lips area and for the eyebrows area based on the fractional displacements of the local markers.

17. The method of claim 1 wherein a first set of markers identifies global motion and a second set of markers identifies local motion of the face.

18. The method of claim 17 wherein the markers comprise at least two colors.

19. The method of claim 18 wherein the two colors are contrasting.

20. The method of claim 19 wherein the colors are black and white.

21. The method of claim 1 wherein the markers comprise two concentric circles of different colors.

22. The method of claim 21 wherein the outer circle has a diameter at least twice the diameter of the inner circle.

23. The method of claim 1 wherein the step of selecting comprises wearing a head-set with markers.

24. The method of claim 23 wherein the head set comprises a strap for a chin.

25. The method of claim 23 wherein the head-set comprises a strap for eyebrows.

26. The method of claim 23 wherein the head-set comprises at least one strap for a skull.

27. The method of claim 1 wherein the acquired 2-D images include at least two views of the face with markers in a neutral state at different orientations.

28. The method of claim 27 wherein the two views are orthogonal.

29. The method of claim 1 wherein the acquired 2-D images comprise front, forehead, chin, angled-right, angled-right-tilted-up, angled-right-tilted-down, angled-left, angled-left-tilted-up, angled-left-tilted-down, full-right-profile, full-right-profile-tilted-up, full-right-profile-tilted-down, full-left-profile, full-left-profile-tilted-up, and full-left-profile-tilted-down views of the face with markers in the neutral state.

30. The method of claim 1 wherein the acquired 2-D images comprise front, forehead, chin, full-right-profile, and full-left-profile views of the face with markers in the neutral state.

31. The method of claim 1 wherein the acquired 2-D images include a plurality of views of the face with markers in at least one action state.

32. The method of claim 31 wherein the action states of the face comprise smiling lips, kissing lips, yawning lips, raised eyebrows, and squeezed eyebrows.

33. The method of claim 31 wherein the acquired 2-D images of the face in an action state include at least two views at different orientations.

34. The method of claim 33 wherein the two views are front and angled-right.

35. The method of claim 1 wherein the step of calculating the 3-D locations of the global and local markers in the neutral state comprises the steps of:

calculating the 3-D locations of the markers to conform to their 2-D locations in the 2-D images of the face in the neutral state under an orthographic projection model;

calculating relative distances of the face to the camera in the 2-D images to conform to the 2-D locations of the markers and their calculated 3-D locations under a perspective projection model;

modifying the 2-D locations of the markers to conform to the calculated relative distances and the 3-D locations under a perspective projection model;

recalculating the 3-D locations of the markers to conform to their modified 2-D locations under an orthographic projection model;

repeating the steps of calculating the relative distances, modifying the 2-D locations, and recalculating the 3-D locations to satisfy a convergence requirement; and

translating and rotating the 3-D locations so that they correspond to a frontal-looking face.

36. A method for tracking motion of an object in a chronologically ordered sequence of 2-D images of the object comprising the steps of:

selecting global and local salient features of the object for tracking by fixing markers to the object;

calculating 3-D locations of the markers at the global and local salient features for a neutral state of the object, and calculating 3-D locations of the local salient features for action states of the object;

calculating the 3-D locations of the global and local markers in a neutral state; and

calculating the 3-D locations of the local markers in each action state by estimating the orientation and position of the face in each 2-D image of the action state to conform to the 3-D and 2-D locations of the global markers under a perspective projection model and calculating the 3-D locations of the local markers to conform to the estimated orientation and position of the face and the 2-D locations of the local markers under a perspective projection model;

predicting 2-D locations of the markers at the global and local salient features in a 2-D image;

detecting 2-D locations of the markers at the global and local salient features in the 2-D image; and

estimating the global and local motion of the object in the 2-D image.

37. The method of claim 36 where the action states of the object define maximum local motions of the object.

38. The method of claim 37 where the method of calculating comprises:

acquiring a plurality of initial 2-D images of the object in neutral state;

acquiring a plurality of initial 2-D images of the object in action states identifying the 2-D locations of the salient features in each initial 2-D image;

estimating the orientation and position of the object in each initial 2-D image to conform to the 2-D locations of the salient features under a perspective projection model; and

calculating 3-D locations of the salient features to conform to the 2-D locations of the salient features under a perspective projection model.

39. The method of claim 36 wherein a first set of markers define global salient features and a second set of markers define local salient features.

40. The method of claim 36 wherein the step of estimating global motion comprises calculating 3-D position and orientation of the object to conform to the 3-D locations of the global markers in the neutral state and the detected 2-D locations of the global markers under a perspective projection model.

41. The method of claim 36 wherein the step of estimating local motion comprise calculating a vector of weights representing fractions of maximum actions of the object conforming to the 3-D locations of local markers in the neutral and action states and the detected 2-D locations of the local markers under a perspective projection model.

42. The method of claim 36 wherein the object is a face.

43. The method of claim 36 wherein the markers comprise concentric circles with two contrasting colors.

44. The method of claim 43 wherein the step of detecting comprises applying elliptical correlation filters in a neighborhood of the markers.