Method for facial animation

• US 10,013,787 B2
• Filed: 12/12/2011
• Issued: 07/03/2018
• Est. Priority Date: 12/12/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Faceshift AG (Apple Inc.)
• Inventors: Weise, Thibaut, Bouaziz, Sofien, Li, Hao, Pauly, Mark
• Primary Examiner(s): Liu, Zhengxi
• Assistant Examiner(s): Isanians, Raffi
• Application Number:
  US13/323,231
• Publication Number:
  US 20130147788A1
• Time in Patent Office: 2,395 Days
• CPC Class Codes: G06F 18/2135 based on approximation criteria, e.g. principal component analysis
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

A method of animating a digital character according to facial expressions of a user, comprising the steps of, (a) obtaining a 2D image and 3D depth map of the face of the user, (b) determining expression parameters for a user expression model so that a facial expression of the user-specific expression model represents the face of the user shown in the 2D image and 3D depth map (c) using the expression parameters and an animation prior to determine animation parameters usable to animate a digital character, wherein the animation prior is a sequence of animation parameters which represent predefined animations of a digital character (d) using the animation parameters to animate a digital character so that the digital character mimics the face of the user.

Claims

1. A method of animating a digital character based on facial expressions of a user, comprising:

obtaining a first series of two-dimensional (2D) images of a face of a user;

obtaining a first series of three-dimensional (3D) depth maps of the face of the user;

determining a set of blendshape weights associated with a generic expression model based on at least some of the first series of 2D images and at least some of the first series of 3D depth maps, the generic expression model representative of a generic person;

identifying expression parameters for a user-specific expression model based on at least some of the set of blendshape weights, the user-specific expression model representative of the face of the user;

tracking the face of the user by decoupling rigid motion of the user from non-rigid motion of the user based on at least some of the first series of 2D images and at least some of the first series of 3D depth maps, wherein the rigid motion represents a movement of the 3D depth map of the face of the user and the non-rigid motion represents a change in expression of the face of the user;

determining animation parameters for a digital character based on the expression parameters, the rigid and non-rigid motions of the user, and an animation prior, the animation prior including a collection of animation parameters of the digital character, the animation prior indicative of a pre-defined animation of the generic expression model; and

animating, based on the animation parameters, the digital character to mimic the face of the user.

2. The method of claim 1, wherein tracking the rigid motion of the user comprises aligning, based on at least some of the first series of 3D depth maps, the user-specific expression model to a position representing a position of the face of the user.

3. The method of claim 2, wherein aligning the user-specific expression model comprises using a pre-segmented template, the pre-segmented template excluding a chin region of the face of the user.

4. The method of claim 1, wherein tracking the non-rigid motion of the user comprises performing an optimization, using the animation prior and the expression parameters, to determine the animation parameters.

5. The method of claim 4, wherein tracking the non-rigid motion of the user comprises avoiding an unrealistic facial expression by regularizing the expression parameters with a dynamic expression prior, the dynamic expression prior based on a set of blendshape animations.

6. The method of claim 1, further comprising converting the expression parameters into a form readable by an animation tool to enable the animation tool to animate the digital character according to the expression parameters.

7. A system for animating a digital character according to facial expressions of a user, comprising:

memory containing a program code;

a display coupled to the memory; and

one or more processors coupled to the memory and the display, the one or more processors configured to execute the program code, the program code configured to cause the one or more processors to;

obtain a first series of two-dimensional (2D) images of a face of a user;

obtain a first series of three-dimensional (3D) depth maps of the face of the user;

determine a set of blendshape weights associated with a generic expression model based on at least some of the first series of 2D images and at least some of the first series of 3D depth maps, the generic expression model representative of a generic person;

identify expression parameters for a user-specific expression model based on at least some of the set of blendshape weights, the user-specific expression model representative of the face of the user;

track the face of the user by decoupling rigid motion of the user from non-rigid motion of the user based on at least some of the first series of 2D images and at least some of the first series of 3D depth maps, wherein the rigid motion represents a movement of the 3D depth map of the face of the user and the non-rigid motion represents a change in expression of the face of the user;

determine animation parameters for a digital character based on the expression parameters, the rigid and non-rigid motions of the user, and an animation prior, the animation prior including a collection of animation parameters of the digital character, the animation prior indicative of a pre-defined animation of the generic expression model; and

animate, based on the animation parameters, the digital character to mimic the face of the user on the display.

8. The system of claim 7, wherein the program code to track the rigid motion of the user comprises program code to align, based on at least some of the first series of 3D depth maps, the user-specific expression model to a position representing a position of the face of the user.

9. The system of claim 8, wherein the program code to align the user-specific expression model comprises program code to use a pre-segmented template, the pre-segmented template excluding a chin region of the face of the user.

10. The system of claim 7, wherein the program code to track the non-rigid motion of the user comprises program code to perform an optimization, using the animation prior and the expression parameters, to determine the animation parameters.

11. The system of claim 10, wherein the program code to track the non-rigid motion of the user comprises program code to prevent an unrealistic facial expression by regularizing the expression parameters with a dynamic expression prior, the dynamic expression prior based on a set of blendshape animations.

12. The system of claim 7, wherein the program code further comprises program code to convert the expression parameters into a form readable by an animation tool to enable the animation tool to animate the digital character according to the expression parameters.

13. A non-transitory program storage device containing instructions that, when executed by a computer system, cause the computer system to:

obtain a first series of two-dimensional (2D) images of a face of a user;

obtain a first series of three-dimensional (3D) depth maps of the face of the user;

determine a set of blendshape weights associated with a generic expression model based on at least some of the first series of 2D images and at least some of the first series of 3D depth maps, the generic expression model representative of a generic person;

identify expression parameters for a user-specific expression model based on at least some of the set of blendshape weights, the user-specific expression model representative of the face of the user;

track the face of the user by decoupling rigid motion of the user from non-rigid motion of the user based on at least some of the first series of 2D images and at least some of the first series of 3D depth maps, wherein the rigid motion represents a movement of the 3D depth map of the face of the user and the non-rigid motion represents a change in expression of the face of the user;

determine animation parameters for a digital character based on the expression parameters, the rigid and non-rigid motions of the user, and an animation prior, the animation prior including a collection of animation parameters of the digital character, the animation prior indicative of a pre-defined animation of the generic expression model; and

animate, based on the animation parameters, the digital character to mimic the face of the user.

14. The non-transitory program storage device of claim 13, wherein the instructions to track the rigid motion of the user comprise instructions to align, based on at least some of the first series of 3D depth maps, the user-specific expression model to a position representing a position of the face of the user.

15. The non-transitory program storage device of claim 14, wherein the instructions to align the user-specific expression model comprise instructions to use a pre-segmented template, the pre-segmented template excluding a chin region of the face of the user.

16. The non-transitory program storage device of claim 13, wherein the instructions to track the non-rigid motion of the user comprise instructions to perform an optimization, using the animation prior and the expression parameters, to determine the animation parameters.

17. The non-transitory program storage device of claim 16, wherein the instructions to track the non-rigid motion of the user comprise instructions to prevent an unrealistic facial expression by regularizing the expression parameters with a dynamic expression prior, the dynamic expression prior based on a set of blendshape animations.

18. The non-transitory program storage device of claim 13, further comprising instructions to convert the expression parameters into a form readable by an animation tool to enable the animation tool to animate the digital character according to the expression parameters.