SYSTEMS AND METHODS FOR CREATING AND DISTRIBUTING MODIFIABLE ANIMATED VIDEO MESSAGES

• US 20160163084A1
• Filed: 02/16/2016
• Published: 06/09/2016
• Est. Priority Date: 03/06/2012
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Corazza, Stefano, Babcock, Daniel J., Pina, Charles, Drouin, Sylvio
• Granted Patent: US 9,626,788 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/205 Re-meshing
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

Systems and methods in accordance with embodiments of the invention enable collaborative creation, transmission, sharing, non-linear exploration, and modification of animated video messages. One embodiment includes a video camera, a processor, a network interface, and storage containing an animated message application, and a 3D character model. In addition, the animated message application configures the processor to: capture a video sequence using the video camera; detect a human face within a sequence of video frames; track changes in human facial expression of a human face detected within a sequence of video frames; map tracked changes in human facial expression to motion data, where the motion data is generated to animate the 3D character model; apply motion data to animate the 3D character model; render an animation of the 3D character model into a file as encoded video; and transmit the encoded video to a remote device via the network interface.

Claims

1. A computer-implemented method for generating animated video messages using video images of human faces, the method comprising:

receiving a sequence of video frames;

detecting a human face within the sequence of the video frames;

tracking, within the sequence of video frames, facial expression changes in the detected human face;

mapping, by at least one processor, the tracked facial expression changes to three-dimensional motion data; and

applying the mapped three-dimensional motion data to a three-dimensional character model to animate a face of the three-dimensional character model.

2. The method of claim 1, further comprising detecting facial expressions from the detected human face within the sequence of video frames.

3. The method of claim 2, further comprising matching the detected facial expressions to a stored facial expression model from a database of stored facial expressions.

4. The method of claim 3, further comprising obtaining, based on matching the detected facial expressions to the stored facial expression model, the three-dimensional mapping motion data for the stored facial expression model.

5. The method of claim 1, wherein the three-dimensional character model comprises data that describes an articulated skeleton, at least one mesh, and skinning weights that indicate how the articulated skeleton deforms the at least one mesh for a given motion.

6. The method of claim 5, wherein the articulated skeleton comprises a plurality of bones configured to rotate in one or more degrees of freedom around joints, and wherein the at least one mesh comprises a plurality of vertices defining a surface having a plurality of polygons.

7. The method of claim 1, wherein detecting the human face within the sequence of the video frames comprises using a cascade of classifiers to detect the human face within the sequence of video frames.

8. The method of claim 1, further comprising obtaining a description of a space of human faces and facial expressions using a training data set containing multiple images of human faces registered to a template image of a human face and multiple images of human facial expressions registered to the same template image of a human face.

9. The method of claim 8, wherein tracking facial expressions in the detected human face comprises matching the detected human face to a point in the space of human faces and facial expressions using the description of a space of human faces and facial expressions.

10. The method of claim 8, further comprising performing principal component analysis of multiple images of human facial expressions registered to the same template image of a human face to define a vector space of human faces and human facial expressions.

11. The method of claim 8, further comprising locating a vector within the space of human faces and human facial expressions that synthesizes a human face that is the closest match to the detected human face in accordance with at least one matching criterion.

12. The method of claim 1, further comprising animating the face of the three-dimensional character model by deforming the mesh of the three-dimensional character using a sequence of facial expressions for the three-dimensional character model based on the applied mapped three-dimensional motion data.

13. A computer-implemented method for generating animated video messages using video images of human faces, the method comprising:

capturing a sequence of video frames;

detecting a facial expression within the sequence of the video frames;

matching, by at least one processor, the detected facial expression to a stored facial expression;

obtaining three-dimensional motion data associated with the stored facial expression that matches the detected facial expression; and

applying the three-dimensional motion data to a three-dimensional character model to animate a face of the three-dimensional character model such that the animation replicates the detected facial expression on the three-dimensional character model.

14. The method of claim 13, further comprising obtaining a description of a space of human faces and facial expressions using a training data set containing multiple images of human faces registered to a template image of a human face and multiple images of human facial expressions registered to the same template image of a human face.

15. The method of claim 14, wherein matching the detected facial expression to the stored facial expression comprises matching the detected human face to a point in the space of human faces and facial expressions using the description of a space of human faces and facial expressions.

16. The method of claim 14, further comprising performing principal component analysis of multiple images of human facial expressions registered to the same template image of a human face to define a vector space of human faces and human facial expressions.

17. The method of claim 14, further comprising locating a vector within the space of human faces and human facial expressions that synthesizes a human face that is the closest match to the detected human face in accordance with at least one matching criterion.

18. A system for generating animated video messages using video images of human faces, the system comprising:

at least one processor; and

at least one non-transitory computer readable storage medium storing instructions that, when executed by the at least one processor, cause the system to;

receive a sequence of video frames;

detect a human face within the sequence of the video frames;

track, within the sequence of video frames, facial expression changes in the detected human face;

map the tracked facial expression changes to three-dimensional motion data; and

apply the mapped three-dimensional motion data to a three-dimensional character model to animate a face of the three-dimensional character model.

19. The system of claim 18, further comprising instructions that, when executed by the at least one processor, cause the system to obtain a description of a space of human faces and facial expressions using a training data set containing multiple images of human faces registered to a template image of a human face and multiple images of human facial expressions registered to the same template image of a human face.

20. The system of claim 19, wherein tracking facial expressions in the detected human face comprises matching the detected human face to a point in the space of human faces and facial expressions using the description of a space of human faces and facial expressions.