Apparatus and method for generating facial animation

• US 8,624,901 B2
• Filed: 02/03/2010
• Issued: 01/07/2014
• Est. Priority Date: 04/09/2009
• Status: Expired due to Fees

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Park, Bong-cheol, Jung, Sung-gun, Kim, Hyo-won, Park, Moon-ho, Oh, Du-na
• Primary Examiner(s): BROOME, SAID A
• Application Number:
  US12/699,229
• Publication Number:
  US 20100259538A1
• Time in Patent Office: 1,434 Days
• Field of Search: None
• US Class Current: 345/473
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings

Abstract

An apparatus and method for generating an animation of a virtual character using video data and a limited number of three-dimensional (3D) key models of a virtual character, are provided. The video data includes a sequence of two-dimensional image frames. The key models are blended together using the calculated weights, and a 3D facial expression most similar to the 2D facial expression included in each image frame may be generated. When the above operations are performed for input video data including a sequence of image frames, a facial animation, which makes facial expressions according to facial expressions included in the video data may be created.

Claims

1. An apparatus for generating facial animation, the apparatus comprising:

a pre-processing unit for selecting an image frame having a facial region that is most similar to a facial region of one of a plurality of key models of facial characters, from a sequence of image frames, each image frame being divided into a plurality of facial regions;

a facial animation generation unit for determining a blending weight for each selected facial region of the key models using the selected facial region and for synthesizing facial character shapes on a facial region-by-facial region basis using the determined blending weight for each facial region of each of the key models; and

a display device for displaying facial animation,

wherein the pre-processing unit compares directions of respective two-dimensional (2D) displacement vectors of vertices included in each key model with directions of respective optical flow vectors of pixels included in each image frame and calculates a similarity between the directions, wherein the 2D displacement vectors are determined based on vertices included in one of the key models, and the optical flow vectors of the pixels are generated by calculating optical flows of the pixels based on pixels included in one of the image frames.

2. The apparatus of claim 1, wherein the pre-processing unit processes each of the plurality of key models and determines a plurality of facial regions for the key models, and sets patches corresponding individually to the facial regions of each of the key models, in each of the image frames.

3. The apparatus of claim 2, wherein the pre-processing unit divides each of the key models into a left eye region, a right eye region, and a mouth region, and sets patches, which correspond to the left eye region, the right eye region, and the mouth region, in each of the image frames.

4. The apparatus of claim 1, wherein the pre-processing unit determines respective three-dimensional (3D) displacement vectors of the vertices included in each key model, based on one of the key models, and projects the 3D displacement vectors onto a 2D plane to generate the 2D displacement vectors of the vertices.

5. The apparatus of claim 1, wherein the pre-processing unit divides each facial region of each key model into a plurality of sub-facial regions and divides each facial region of each image frame into a plurality of sub-facial regions and calculates a similarity between a direction of an average vector representing the average of 2D displacement vectors in each sub-facial region of each key model and a direction of an average vector representing the average of optical flow vectors in a corresponding sub-facial region of each image frame.

6. The apparatus of claim 5, wherein the pre-processing unit divides each facial region of each key model and of each image frame into quadrants.

7. The apparatus of claim 1, wherein the pre-processing unit sets an optical flow vector of the facial region of the image frame which is most similar to a facial region of a key model, as a seed, clusters optical flow vectors of the facial regions of the other image frames using the optical flow vector set as the seed, and determines an image frame having an optical flow vector, which corresponds to the average of all optical vectors in each cluster, as an image frame having a facial region most similar to a facial region of a key model in the cluster.

8. The apparatus of claim 7, wherein the pre-processing unit groups the optical flow vectors of the image frames into a number of clusters equal to the number of key models.

9. The apparatus of claim 6, wherein the facial animation generation unit uses a four-dimensional (4D) vector, which indicates a direction of an optical flow vector in each quadrant of each facial region, as a parameter to determine the blending weight for each facial region of each key model.

11. The apparatus of claim 1, wherein the facial animation generation unit generates a whole facial character by generating a facial character for each facial region using the blending weight for each facial region of each key model and by synthesizing the generated facial characters.

12. The apparatus of claim 1, wherein when video data about the same person as the one in a facial image included in the image frames of the pre-processed video is input, the facial animation generation unit generates facial animation using representative parameter information corresponding to the facial region of the image frame, which is most similar to a facial region of a key model.

19. The apparatus of claim 1, wherein the key models are represented as spring-mass networks and the position of mass points may vary according to a facial expression of the key models.

10. An apparatus for generating facial animation, the apparatus comprising:

a pre-processing unit for selecting an image frame having a facial region that is most similar to a facial region of one of a plurality of key models of facial characters, from a sequence of image frames, each image frame being divided into a plurality of facial regions;

a facial animation generation unit for determining a blending weight for each selected facial region of the key models using the selected facial region and for synthesizing facial character shapes on a facial region-by-facial region basis using the determined blending weight for each facial region of each of the key models; and

a display device for displaying facial animation,

wherein the facial animation generation unit determines the blending weight for each facial region of each key model such that the determined blending weight is inversely proportional to a distance of a representative parameter value corresponding to each key model from the parameter of the image frame.

13. A method of generating facial animation based on a video, the method comprising:

selecting an image frame having a facial region that is most similar to a facial region of one of a plurality of key models of facial characters, from a sequence of image frames;

determining a blending weight for each facial region of each of the key model using the selected image frame;

generating facial animation by synthesizing facial characters on a facial region-by-facial region basis using the determined blending weight for each facial region of each key model; and

displaying the facial animation on a display device,

wherein each of the key models is divided into a plurality of facial regions, and each of the image frames is divided into a plurality of facial regions, wherein the selecting of the image frame having a facial region most similar to a facial region of one of a key model further comprises comparing directions of respective 2D displacement vectors of vertices included in each key model with directions of respective optical flow vectors of pixels included in each image frame and calculating a similarity between the directions, wherein the 2D displacement vectors are determined based on vertices included in one of the key models, and the optical flow vectors of the pixels are generated by calculating optical flows of the pixels based on pixels included in one of the image frames.

14. The method of claim 13, further comprising setting patches in each image frame, which are regions corresponding to the facial regions of each of the key models, in video data comprised of the image frames.

15. The method of claim 13, wherein each of the key models is a 3D shape, and the 2D displacement vectors of the vertices are generated by determining respective 3D displacement vectors of the vertices included in each key model based on one of the key models and projecting the 3D displacement vectors onto a 2D plane.

16. The method of claim 13, wherein the comparing of the directions and calculating of the similarity comprises:

dividing each facial region of each key model into a plurality of sub-facial regions and dividing each facial region of each image frame into a plurality of sub-facial regions; and

calculating a similarity between a direction of an average vector representing the average of 2D displacement vectors in each sub-facial region of each key model and a direction of an average vector representing the average of optical flow vectors in a corresponding sub-facial region of each image frame.

17. The method of claim 13, wherein a parameter used to determine the blending weight for each facial region of each key model is a 4D vector.

18. The method of claim 13, further comprising generating a complete facial character by generating the facial character for each facial region using the blending weight for each facial region of each key model and synthesizing the generated facial characters.

20. An apparatus for generating facial animation, the apparatus comprising:

a pre-processing unit for selecting an image frame having a facial region that is most similar to a facial region of one of a plurality of key models of facial characters, from a sequence of image frames, each image frame being divided into a plurality of facial regions;

a facial animation generation unit for determining a blending weight for each selected facial region of the key models using the selected facial region and for synthesizing facial character shapes on a facial region-by-facial region basis using the determined blending weight for each facial region of each of the key models; and

a display device for displaying facial animation,

wherein a key model region divider may extract mass points of each of the key models when a change in length of a spring is greater than a predetermined threshold value compared with a reference key model, and may divide the extracted mass points into a plurality of groups according to coherence in motion.