ONLINE MODELING FOR REAL-TIME FACIAL ANIMATION

• US 20140362091A1
• Filed: 06/07/2013
• Published: 12/11/2014
• Est. Priority Date: 06/07/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Ecole polytechnique fãdãrale de lausanne epfl (Schweizerische Eidgenossenschaft)
• Inventors: PAULY, MARK, BOUAZIZ, SOFIEN
• Granted Patent: US 9,378,576 B2
• Time in Patent Office: Days
• US Class Current: 345/473
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

Embodiments relate to a method for real-time facial animation, and a processing device for real-time facial animation. The method includes providing a dynamic expression model, receiving tracking data corresponding to a facial expression of a user, estimating tracking parameters based on the dynamic expression model and the tracking data, and refining the dynamic expression model based on the tracking data and estimated tracking parameters. The method may further include generating a graphical representation corresponding to the facial expression of the user based on the tracking parameters. Embodiments pertain to a real-time facial animation system.

Claims

1. A method for real-time facial animation, comprising:

providing a dynamic expression model;

receiving tracking data corresponding to a facial expression of a user;

estimating tracking parameters based on the dynamic expression model and the tracking data; and

refining the dynamic expression model based on the tracking data and the estimated tracking parameters to produce a refined dynamic expression model.

2. The method of claim 1, wherein estimating tracking parameters and refining the dynamic expression model are performed in real-time.

3. The method of claim 1, wherein estimating tracking parameters is performed in a first stage and refining the dynamic expression model is performed in a second stage, wherein the first stage and the second stage are iteratively repeated.

4. The method of claim 1, further comprising generating a graphical representation corresponding to the facial expression of the user based on the tracking parameters.

5. The method of claim 1, further comprising:

receiving further tracking data corresponding to a further facial expression of the user,

estimating updated tracking parameters based on the refined dynamic expression model and the further tracking data, and

generating a graphical representation corresponding to the further facial expression of the user based on the updated tracking parameters.

6. The method of claim 1, wherein the dynamic expression model includes a plurality of blendshapes and the tracking parameters include a corresponding plurality of weights for the plurality of blendshapes.

7. The method of claim 6, wherein the plurality of blendshapes includes a blendshape b₀ representing a neutral facial expression and the dynamic expression model further includes an identity model, wherein the method further comprises matching the blendshape b₀ representing the neutral facial expression to a neutral expression of the user based on the identity model.

8. The method of claim 7, wherein the plurality of blendshapes further includes one or more blendshapes b_i, each representing a corresponding facial expression of one or more additional facial expressions, wherein the dynamic expression model further includes a template blendshape model, wherein the method further comprises approximating the one or more blendshapes b_i based on the template blendshape model and the blendshape b₀ representing the neutral facial expression.

9. The method of claim 6, wherein the dynamic expression model further includes one or more corrective deformation fields, wherein the method further comprises applying at least one of the one or more corrective deformation fields to each of the plurality of blendshapes.

10. The method of claim 1, further comprising:

receiving neutral facial expression tracking data corresponding to a neutral facial expression of the user; and

initializing the dynamic expression model using the tracking data corresponding to the neutral facial expression of the user.

11. The method of claim 8, further comprising parameterizing the one or more blendshapes b_i as b_i=T*_ib₀+Ez_i, wherein T*_i is an expression transfer operator derived from the template blendshape model, and Ez_i is a corrective deformation field for the blendshape b_i.

12. The method of claim 1, wherein refining the dynamic expression model based on the tracking data and the estimated tracking parameters comprises refining the dynamic expression model based on the tracking data, the estimated tracking parameters, and aggregated tracking data, wherein the aggregated tracking data is based on additional received tracking data corresponding to one or more past facial expressions of the user.

13. The method of claim 12, further comprising aggregating the additional received tracking data corresponding to one or more past facial expressions of the user subject to a decay over time to produce the aggregated tracking data.

14. The method of claim 8, wherein the plurality of blendshapes has n blendshapes, wherein refining the dynamic expression model includes:

determining a coverage coefficient σ

_i, for i=0, 1, . . . , n−

1, for each blendshape b_i, for i=0, 1, . . . , n−

1, of the dynamic expression model, wherein the coverage coefficient σ

_i is indicative of the applicability of past tracking data corresponding to past facial expressions of the user for the blendshape b_i; and

refining each blendshape b_i, for i=0, 1, . . . , n−

1, having a coverage coefficient σ

_i, for i=0, 1, . . . , n−

1, below a pre-determined threshold.

15. The method of claim 4, further comprising:

automatically creating a virtual avatar of the user based on the dynamic expression model; and

using the virtual avatar for generating the graphical representation corresponding to the facial expression of the user.

16. The method of claim 4, wherein the graphical representation corresponding to the facial expression of the user is generated based on one or more blendshapes of the plurality of blendshapes representing a virtual avatar.

17. A processing device, comprising:

an input interface configured to receive tracking data corresponding to a facial expression of a user;

a memory configured to store a dynamic expression model; and

a processing component coupled to the input interface and the memory, wherein the processing component is configured to;

estimate tracking parameters based on the dynamic expression model and the tracking data, and

refine the dynamic expression model based on the tracking data and the estimated tracking parameters to produce a refined dynamic expression model.

18. The device of claim 17, wherein the processing component is further configured to estimate the tracking parameters and refine the dynamic expression model in real-time.

19. The device of claim 17, wherein the processing component is further configured to generate a graphical representation corresponding to the facial expression of the user based on the tracking parameters.

20. The device of claim 17, wherein the dynamic expression model includes a plurality of blendshapes and the tracking parameters include a corresponding plurality of weights for the plurality of blendshapes.

21. The device of claim 20, wherein the plurality of blendshapes includes a blendshape b₀ representing a neutral facial expression and the dynamic expression model further includes an identity model, wherein the processing component is further configured to match the blendshape b₀ representing the neutral facial expression to a neutral expression of the user based on the identity model.

22. The device of claim 21, wherein the plurality of blendshapes further includes one or more blendshapes b_i, each representing a corresponding facial expression of one or more additional facial expressions, wherein the dynamic expression model further includes a template blendshape model, wherein the processing component is further configured to approximate the one or more blendshapes b_i based on the template blendshape model and the blendshape b₀ representing the neutral facial expression.

23. The device of claim 20, wherein the dynamic expression model further includes one or more corrective deformation fields, wherein the processing component is further configured to apply at least one of the one or more corrective deformation fields to each of the plurality of blendshapes.

24. The device of claim 17, wherein the processing component is further configured to refine the dynamic expression model based on the tracking data, the estimated tracking parameters, and aggregated tracking data, wherein the aggregated tracking data is based on additional received tracking data corresponding to one or more past facial expressions of the user, wherein the aggregated tracking data is produced by aggregating the additional received tracking data corresponding to one or more past facial expressions of the user subject to a decay over time.

25. The device of claim 22, wherein the plurality of blendshapes has n blendshapes, wherein, in order to refine the dynamic expression model, the processing component is further configured to:

determine a coverage coefficient σ

_i, for i=0, 1, . . . , n−

1, for each blendshape b_i, for i=0, 1, . . . , n−

1, of the dynamic expression model, wherein the coverage coefficient σ

_i for i=0, 1, . . . , n−

1, is indicative of the applicability of past tracking data corresponding to past facial expressions for the blendshape b_i, for i=0, 1, . . . , n−

1, and

only refine each blendshape b_i, for i=0, 1, . . . , n−

1, having a coverage coefficient σ

_i, for i=0, 1, . . . , n−

1, below a pre-determined threshold.

26. A real-time facial animation system, comprising:

a camera device, wherein the camera device is configured to generate tracking data corresponding to a facial expression of a user; and

a processing device, wherein the processing device comprises;

an input interface, wherein the input interface is coupled to the camera device and configured to receive the tracking data,

a memory, wherein the memory is configured to store a dynamic expression model, wherein the dynamic expression model includes a plurality of blendshapes, and

a processing component, wherein the processing component is coupled to the input interface and the memory, wherein the processing component is configured to estimate a corresponding plurality of weights for the plurality of blendshapes of the dynamic expression model based on the tracking data, wherein the processing component is configured to generate a graphical representation corresponding to the facial expression of the user based on the plurality of weights, wherein the processing component is configured to refine the dynamic expression model based on the tracking data and the plurality of weights for the blendshapes.

27. The system of claim 26, wherein the camera device is configured to generate tracking data including video data and depth information.