Systems and methods for creating animations using human faces

• US 9,626,788 B2
• Filed: 02/16/2016
• Issued: 04/18/2017
• Est. Priority Date: 03/06/2012
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Babcock, Daniel J., Corazza, Stefano, Pina, Charles, Drouin, Sylvio
• Primary Examiner(s): Gray, Ryan M
• Application Number:
  US15/044,970
• Publication Number:
  US 20160163084A1
• Time in Patent Office: 427 Days
• Field of Search: None
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/205 Re-meshing
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

Systems and methods in accordance with embodiments of the invention enable collaborative creation, transmission, sharing, non-linear exploration, and modification of animated video messages. One embodiment includes a video camera, a processor, a network interface, and storage containing an animated message application, and a 3D character model. In addition, the animated message application configures the processor to: capture a video sequence using the video camera; detect a human face within a sequence of video frames; track changes in human facial expression of a human face detected within a sequence of video frames; map tracked changes in human facial expression to motion data, where the motion data is generated to animate the 3D character model; apply motion data to animate the 3D character model; render an animation of the 3D character model into a file as encoded video; and transmit the encoded video to a remote device via the network interface.

Claims

1. A computer-implemented method for generating animations messages using video images of human faces, the method comprising:

receiving a sequence of video frames;

detecting a human face within the sequence of the video frames;

tracking, within the sequence of video frames, facial expression changes in the detected human face;

identifying, for each of the tracked facial expression changes, a facial expression from a training set of a plurality of different facial expressions and a plurality of different user faces that most closely corresponds to the tracked facial expression change;

mapping, by at least one processor, each of the identified facial expressions to three-dimensional motion data such that the identified facial expressions corresponds with the tracked facial expression changes from the sequence of video frames; and

applying the three-dimensional motion data to a three-dimensional character model to animate a face of the three-dimensional character model.

2. The method of claim 1, further comprising detecting human facial expressions from the detected human face within the sequence of video frames.

3. The method of claim 2, wherein identifying the facial expression further comprises matching the tracked facial expression change to a facial expression model from the training set of the plurality of different facial expressions and the plurality of different user faces that is stored in a database of facial expressions.

4. The method of claim 3, further comprising obtaining, based on matching the tracked facial expression change to the facial expression model, the three-dimensional motion data for the facial expression model.

5. The method of claim 1, wherein the three-dimensional character model comprises data that describes an articulated skeleton, at least one mesh, and skinning weights that indicate how the articulated skeleton deforms the at least one mesh for a given motion.

6. The method of claim 5, wherein the articulated skeleton comprises a plurality of bones configured to rotate in one or more degrees of freedom around joints, and wherein the at least one mesh comprises a plurality of vertices defining a surface having a plurality of polygons.

7. The method of claim 1, wherein detecting the human face within the sequence of the video frames comprises using a cascade of classifiers to detect the human face within the sequence of video frames.

8. The method of claim 1, wherein identifying the facial expression further comprises:

obtaining a feature vector of the tracked facial expression change within a principal component analysis (PCA) space; and

identifying, in the PCA space, the facial expression from the training set of the plurality of different facial expressions and the plurality of different user faces that has a feature vector that most closely matched the obtained feature vector of the tracked facial expression.

9. The method of claim 1, wherein tracking facial expression changes comprises tracking landmark points, edges, and colors of the detected human face within the sequence of the video frames.

10. The method of claim 1, wherein identifying the facial expression further comprises performing principal component analysis of the training set of the plurality of different facial expressions and the plurality of different user faces to identify the facial expression that most closely corresponds to the tracked facial expression change.

11. The method of claim 1, wherein the training set of the plurality of different facial expressions and the plurality of different user faces comprise a set of three-dimensional images that include additional metadata including depth maps.

12. The method of claim 1, further comprising animating the face of the three-dimensional character model by deforming the mesh of the three-dimensional character using a sequence of facial expressions for the three-dimensional character model based on the applied three-dimensional motion data.

13. A computer-implemented method for generating animations using video images of human faces, the method comprising:

capturing a sequence of video frames;

detecting a facial expression within the sequence of the video frames;

matching, by at least one processor, the detected facial expression to a stored facial expression from a training set of a plurality of different facial expressions and a plurality of different user faces that most closely corresponds to the detected facial expression;

obtaining three-dimensional motion data associated with the stored facial expression that matches the detected facial expression; and

applying the three-dimensional motion data to a three-dimensional character model to animate a face of the three-dimensional character model such that the animation replicates the detected facial expression on the three-dimensional character model.

14. The method of claim 13, wherein the training set of the plurality of different facial expressions and the plurality of different user faces is stored in a database of stored facial expressions and comprises a set of three-dimensional images that include additional metadata including depth maps.

15. The method of claim 14, wherein matching the detected facial expression to the stored facial expression comprises matching the detected facial expression to the stored facial expression from the training set of the plurality of different facial expressions and the plurality of different user faces by comparing feature vectors between the detected facial expression and facial expressions in the training set of the plurality of different facial expressions and the plurality of different user faces.

16. The method of claim 14, wherein matching the detected facial expression to the stored facial expression comprises further performing principal component analysis of the training set of the plurality of different facial expressions and the plurality of different user faces to identify the facial expression that most closely corresponds to the detected facial expression.

17. The method of claim 14, further comprising locating a vector within the training set of the plurality of different facial expressions and the plurality of different user faces that is the closest match to a vector representing the detected face.

18. A system for generating animations using video images of human faces, the system comprising:

at least one processor; and

at least one non-transitory computer readable storage medium storing instructions that, when executed by the at least one processor, cause the system to;

receive a sequence of video frames;

detect a human face within the sequence of the video frames;

track, within the sequence of video frames, facial expression changes in the detected human face;

identify, for each of the tracked facial expression changes, a facial expression from a training set of a plurality of different facial expressions and a plurality of different user faces that most closely corresponds to the tracked facial expression change;

map each identified facial expression to three-dimensional motion data such that the identified facial expressions corresponds with the tracked facial expression changes from the sequence of video frames; and

apply the three-dimensional motion data to a three-dimensional character model to animate a face of the three-dimensional character model.

19. The system of claim 18, wherein the instructions, when executed by the at least one processor, cause the system to identify a facial expression by:

obtaining a feature vector of the tracked facial expression change within a principal component analysis (PCA) space; and

identifying, in the PCA space, the facial expression from the training set of the plurality of different facial expressions and the plurality of different user faces that has a feature vector that most closely matched the obtained feature vector of the tracked facial expression.

20. The system of claim 19, wherein the instructions, when executed by the at least one processor, that cause the system to track facial expression changes in the detected human face by tracking landmark points, edges, and colors of the detected human face within the sequence of the video frames.