METHOD FOR FACIAL ANIMATION

• US 20130147788A1
• Filed: 12/12/2011
• Published: 06/13/2013
• Est. Priority Date: 12/12/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Faceshift GmbH (Apple Inc.)
• Inventors: WEISE, Thibaut, Bouaziz, Sofien, Ll, Hao, Pauly, Mark
• Granted Patent: US 10,013,787 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06F 18/2135 based on approximation criteria, e.g. principal component analysis
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

A method of animating a digital character according to facial expressions of a user, comprising the steps of, (a) obtaining a 2D image and 3D depth map of the face of the user, (b) determining expression parameters for a user expression model so that a facial expression of the user-specific expression model represents the face of the user shown in the 2D image and 3D depth map (c) using the expression parameters and an animation prior to determine animation parameters usable to animate a digital character, wherein the animation prior is a sequence of animation parameters which represent predefined animations of a digital character (d) using the animation parameters to animate a digital character so that the digital character mimics the face of the user.

Claims

1. A method of animating a digital character according to facial expressions of a user, comprising the steps of, (a) obtaining a 2D image and 3D depth map of the face of the user;

(b) determining expression parameters for a user expression model so that a facial expression of the user-specific expression model represents the face of the user shown in the 2D image and 3D depth map;

(c) using the expression parameters and an animation prior to determine animation parameters usable to animate a digital character, wherein the animation prior is a sequence of animation parameters which represent predefined animations of a digital character;

(d) using the animation parameters to animate a digital character so that the digital character mimics the face of the user.

2. The method according to claim 1 wherein the step of using the expression parameters and an animation prior to determine animation parameters usable to animate a digital character, comprises performing an optimisation calculation using the expression parameters and animation prior.

3. The method according to claim 1 further comprising, repeating steps (a)-(d) so that the face of the user is tracked over a period of time, and the digital character mimics the face of the user over that period of time.

4. The method according to claim 3 wherein the method comprises the steps of decoupling rigid motion of a user from non-rigid motion of a user and tracking rigid motion of a user and non-rigid motion of a user independently, wherein rigid motion is a movement of the position of the face of a user and non-rigid motion is a change in expression of the face of a user.

5. The method according to claim 4 wherein the step of tracking rigid motion of a user, comprises aligning the 3D depth map of the face of the user with a user expression model from the preceding iteration of the method, so that the position of the user-specific expression model is moved to a position which best represents the position of the face of the user shown in the 3D depth map.

6. The method according to claim 5 wherein the step of tracking rigid motion comprises the step of using a pre-segmented template that excludes a chin region to stabilise alignment of the 3D depth map with the user-specific expression model.

7. The method according to claim 4 wherein the step of tracking rigid motion comprises the step of filtering using a temporal filter to reduce high frequency flickering of tracking parameters.

8. The method according to claim 4, wherein the step of tracking non-rigid motion of a user comprises, for each 2D image and 3D depth map in a sequence of 2D images and 3D depth maps, performing a single optimization calculation using the expression parameters and the animation prior, to determine animation parameters.

9. The method according to claim 8 wherein the optimization calculation comprises a maximum a posterior estimation.

10. The method according to claim 7 wherein the step of tracking non-rigid motion of a user comprises the step of regularizing the expression parameters with a dynamic expression prior computed, to prevent unrealistic face poses.

11. The method according to claim 8 wherein the single optimization is performed using an iterative gradient solver.

12. The method according to claim 1 further comprising the step of mapping the animation parameters into a form which is readable by an animation tool, so that animation tool can animate the digital character according to the animation parameters.

13. The method according to claim 1 wherein the method further comprises a calibration step wherein the user-specific expression model is generated;

wherein the calibration step comprises the steps of, obtaining a 2D images and 3D depth maps of a face of a user for a plurality of predefined facial expressions;

determining a set of model parameters which represent the face of the user shown in the 2D images and 3D depth maps;

using the model parameters to modify a generic expression model to provide a user-specific expression model.

14. The method according to claim 1 further comprising the step of generating said animation prior, wherein the step of generating said animation prior comprises, animating a generic model using an animation tool, and storing in a memory a sequence of expression parameters which represent the animation of the generic model, wherein the stored expression parameters define the animation prior.

15. The method according to claim 1 further comprising the step of generating said animation priors, wherein the step of generating said animation priors comprises, retrieving a sequence of expression parameters from another digital character animation system wherein the sequence of expression parameters represent animations of a digital character.

16. A computer medium comprising program code to be executed by a signal processing device, said program code being configured to carry out the method of claim 1 when said program code is executed by said signal processing device.