Methods and systems for animating facial expressions

• US 10,198,845 B1
• Filed: 05/29/2018
• Issued: 02/05/2019
• Est. Priority Date: 05/29/2018
• Status: Active Grant

• Current Assignee: LoomAi, Inc.
• Original Assignee: LoomAi, Inc.
• Inventors: Bhat, Kiran, Ramasubramanian, Mahesh, Palleschi, Michael, Johnson, Andrew A., Sachs, Ian
• Primary Examiner(s): Nguyen, Phu K
• Application Number:
  US15/992,114
• Time in Patent Office: 252 Days
• Field of Search: 345419
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/175 Static expression
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

Systems and methods for animating expressions of 3D models from captured images of a user'"'"'s face in accordance with various embodiments of the invention are disclosed. In many embodiments, expressions are identified based on landmarks from images of a user'"'"'s face. In certain embodiments, weights for morph targets of a 3D model are calculated based on identified landmarks and/or weights for predefined facial expressions to animate expressions for the 3D model.

Claims

1. A system for generating animations for a 3D model, the system comprising:

a set of one or more processors;

memory readable by the set of processors; and

instructions stored in the memory that when read by the set of processors directs the set of processors to;

identify a first set of landmarks from a first set of one or more images of a user'"'"'s face;

identify a neutral frame based on the identified set of landmarks;

identify a second set of landmarks from a second set of one or more images of the user'"'"'s face;

classify a facial expression of the user'"'"'s face in the second set of images based on the identified neutral frame and the second set of landmarks;

identify a set of one or more predefined expression weights based on the facial expression; and

calculate a set of final morph target weights from the predefined expression weights and the second set of landmarks based on the second set of images, wherein the 3D model is animated based on the calculated set of final morph target weights for morph targets of the 3D model.

2. The system of claim 1, wherein identifying the neutral frame comprises:

classifying a facial expression of the user'"'"'s face in the first set of images;

adjusting the first set of landmarks for the classified expression to an estimated neutral expression; and

setting the adjusted set of landmarks as a neutral expression.

3. The system of claim 2, wherein classifying a facial expression comprises identifying an expression state based on the identified set of landmarks.

4. The system of claim 2, wherein adjusting the first set of landmarks comprises:

identifying an expected ratio between two sets of facial landmarks for the classified facial expression; and

adjusting the first set of landmarks based on a difference between the expected ratio for the classified facial expression and an expected ratio for a neutral facial expression.

5. The system of claim 1, wherein the first set of landmarks are identified using at least one of a mnemonic descent method (MDM) deep learning approach and an ensemble of regression trees (ERT) approach.

6. The system of claim 1, wherein calculating the set of final morph target weights comprises:

calculating a set of initial morph target weights for the second set of images based on the second set of landmarks; and

blending the set of initial morph target weights and the predefined expression weights to compute the set of final morph target weights.

7. The system of claim 1, wherein the morph targets of the 3D model comprise a set of one or more base shapes and a set of one or more corrective shapes.

8. The system of claim 1, wherein the morph targets of the 3D model are stored in a standardized model file format.

9. The system of claim 1, wherein the instructions further direct the set of processors to adjust the identified neutral frame based on second set of images.

10. The system of claim 1, wherein the instructions further direct the set of processors to calculate a second set of final morph target weights based on a third set of one or more images, wherein the transition between the first set of final morph target weights and the second set of final morph target weights is based on a linear function to control the rate of morphing between different expressions.

11. A method for generating animations for a 3D model, the method comprising:

identifying a first set of landmarks from a first set of one or more images of a user'"'"'s face;

identifying a neutral frame based on the identified set of landmarks;

identifying a second set of landmarks from a second set of one or more images of the user'"'"'s face;

classifying a facial expression of the user'"'"'s face in the second set of images based on the identified neutral frame and the second set of landmarks;

identifying a set of one or more predefined expression weights based on the facial expression; and

calculating a set of final morph target weights from the predefined expression weights and the second set of landmarks based on the second set of images, wherein the 3D model is animated based on the calculated set of final morph target weights for morph targets of the 3D model.

12. The method of claim 11, wherein identifying the neutral frame comprises:

classifying a facial expression of the user'"'"'s face in the first set of images;

adjusting the first set of landmarks for the classified expression to an estimated neutral expression; and

setting the adjusted set of landmarks as a neutral expression.

13. The method of claim 12, wherein classifying a facial expression comprises identifying an expression state based on the identified set of landmarks.

14. The method of claim 12, wherein adjusting the first set of landmarks comprises:

identifying an expected ratio between two sets of facial landmarks for the classified facial expression; and

adjusting the first set of landmarks based on a difference between the expected ratio for the classified facial expression and an expected ratio for a neutral facial expression.

15. The method of claim 11, wherein the first set of landmarks are identified using at least one of a mnemonic descent method (MDM) deep learning approach and an ensemble of regression trees (ERT) approach.

16. The method of claim 11, wherein calculating the set of final morph target weights comprises:

calculating a set of initial morph target weights for the second set of images based on the second set of landmarks; and

blending the set of initial morph target weights and the predefined expression weights to compute the set of final morph target weights.

17. The method of claim 11, wherein the morph targets of the 3D model comprise a set of one or more base shapes and a set of one or more corrective shapes.

18. The method of claim 11, wherein the morph targets of the 3D model are stored in a standardized model file format.

19. The method of claim 11 further comprising adjusting the identified neutral frame based on second set of images.

20. The method of claim 11 further comprising calculating a second set of final morph target weights based on a third set of one or more images, wherein the transition between the first set of final morph target weights and the second set of final morph target weights is based on a linear function to control the rate of morphing between different expressions.