Photo-realistic synthesis of three dimensional animation with facial features synchronized with speech

• US 9,613,450 B2
• Filed: 05/03/2011
• Issued: 04/04/2017
• Est. Priority Date: 05/03/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wang, Lijuan, Soong, Frank, Huo, Qiang, Zhang, Zhengyou
• Primary Examiner(s): Godbold, Douglas
• Assistant Examiner(s): Villena, Mark
• Application Number:
  US13/099,387
• Publication Number:
  US 20120280974A1
• Time in Patent Office: 2,163 Days
• Field of Search: 704235, 704270, 382118, 382203, 345473
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads
• CPC Class Codes: G10L 21/10 Transforming into visible information

Abstract

Dynamic texture mapping is used to create a photorealistic three dimensional animation of an individual with facial features synchronized with desired speech. Audiovisual data of an individual reading a known script is obtained and stored in an audio library and an image library. The audiovisual data is processed to extract feature vectors used to train a statistical model. An input audio feature vector corresponding to desired speech with which the animation will be synchronized is provided. The statistical model is used to generate a trajectory of visual feature vectors that corresponds to the input audio feature vector. These visual feature vectors are used to identify a matching image sequence from the image library. The resulting sequence of images, concatenated from the image library, provides a photorealistic image sequence with facial features, such as lip movements, synchronized with the desired speech. This image sequence is applied to the three-dimensional model.

Claims

1. A computer-implemented method for generating photo-realistic facial animation synchronized with speech, comprising:

storing, in a computer storage device, a statistical model of audiovisual data over time, based on acoustic feature vectors from actual audio data and visual feature vectors of lips images extracted from real sample images of a head and facial features of an individual during a set of utterances by the individual;

storing, in an image library, the real sample images of the individual'"'"'s head and facial features during the set of utterances, including storing for each of the stored real sample images the visual feature vectors obtained from the lips image extracted from the real sample image as used to generate the statistical model;

receiving an input set of acoustic feature vectors for the speech with which the facial animation is to be synchronized;

using a computer processor, applying the received input set of acoustic feature vectors to the statistical model, the statistical model thereby generating a visual feature vector sequence;

selecting, using a computer processor, a sequence of real sample images of the individual'"'"'s head and facial features from the image library, such that the selected sequence matches the visual feature vector sequence generated using the statistical model by comparing visual feature vectors in the visual feature vector sequence with visual feature vectors associated with the real sample images in the image library; and

using a computer processor, applying the selected sequence of real sample images to the three dimensional model of a head to provide the photo-realistic facial animation synchronized with the speech.

2. The computer-implemented method of claim 1, further comprising generating the statistical model, wherein generating the statistical model comprises:

obtaining actual audiovisual data including a plurality of samples including real sample images of the individual'"'"'s facial features for a set of utterances;

extracting the acoustic feature vectors and the visual feature vectors for each sample of the audiovisual data; and

training the statistical model using the acoustic feature vectors and the visual feature vectors.

3. The computer-implemented method of claim 1, wherein generating the visual feature vector sequence comprises maximizing a likelihood function with respect to the input acoustic feature vectors and the statistical model.

4. The computer-implemented method of claim 1, wherein selecting the sequence of real sample images comprises selecting a set of real sample images that minimizes a cost function.

5. The computer-implemented method of claim 4, wherein the cost function comprises a target cost indicative of a difference between a visual feature vector in the generated visual feature vector sequence and a visual feature vector related to a real sample image.

6. The computer-implemented method of claim 5, wherein the cost function comprises a concatenation cost indicative of a difference between adjacent real sample images in the selected sequence of real sample images.

7. The computer-implemented method of claim 1, wherein selecting the sequence of real sample images from the image library comprises identifying a sequence of real sample images from the image library that matches the generated visual feature vector sequence based on both a target cost and a concatenation cost.

8. The computer-implemented method of claim 1, wherein applying the selected sequence of real sample images comprises:

generating, using a computer processor, a sequence of images of the individual'"'"'s head and facial features from the selected sequence of real sample images;

accessing an animated three-dimensional model of a head of the individual comprising a plurality of frames corresponding to the generated sequence of images; and

using a computer processor, applying the generated sequence of images to the three dimensional model as a texture, such that different frames of the animated three-dimensional model are textured by different images of the generated sequence of images, to provide the photo-realistic facial animation synchronized with the speech.

9. A computer system for generating photo-realistic facial animation synchronized with speech, comprising:

a computer storage device storing a statistical model of audiovisual data over time, based on acoustic feature vectors from actual audio data and visual feature vectors of lips images extracted from real sample images of a head and facial features of an individual during a set of utterances by the individual;

an image library storing real sample images of the individual'"'"'s head and facial features during the set of utterances, the image library further storing for each of the stored real sample images the visual feature vectors obtained from the lips image extracted from the real sample image as used to generate the statistical model;

a synthesis module having an input for receiving an input set of feature vectors for speech with which the facial animation is to be synchronized, and providing as an output a visual feature vector sequence corresponding to the input set of feature vectors according to the statistical model;

an image selection module having an input for receiving the visual feature vector sequence from the output of the synthesis module, and accessing the image library using the received visual feature vector sequence to generate an output providing a sequence of real sample images of the individual'"'"'s head and facial features from the image library having visual feature vectors that match the visual feature vectors in the visual feature vector sequence received from the synthesis module by comparing visual feature vectors in the visual feature vector sequence with visual feature vectors associated with the real sample images in the image library; and

an animation module having an input for receiving a three dimensional model of a head and the sequence of real sample images from the image selection module, and an output providing the facial animation synchronized with the speech.

10. The computer system of claim 9, further comprising:

a training module having an input receiving acoustic feature vectors and visual feature vectors from the audiovisual data of an individual'"'"'s facial features during the set of utterances and providing as an output a statistical model of the audiovisual data over time.

11. The computer system of claim 10, wherein the training module comprises:

a feature extraction module having an input for receiving the audiovisual data and providing an output including the acoustic feature vectors and the visual feature vectors corresponding to each sample of the audiovisual data; and

a statistical model training module having an input for receiving the acoustic feature vectors and the visual feature vectors and providing as an output the statistical model.

12. The computer system of claim 9, wherein the synthesis module implements a maximum likelihood function with respect to the input acoustic feature vectors and the statistical model.

13. The computer system of claim 9, wherein the image selection module implements a cost function and identifies a set of real sample images that minimizes the cost function.

14. The computer system of claim 13, wherein the cost function comprises a target cost indicative of a difference between a visual feature vector in the visual feature vector sequence and a visual feature vector related to a real sample image.

15. The computer system of claim 14, wherein the cost function comprises a concatenation cost indicative of a difference between adjacent real sample images in the sequence of real sample images.

16. The computer system of claim 9, wherein the image selection module accesses the image library using the visual feature vector sequence to identify a sequence of real sample images from the image library that matches the visual feature vector sequence based on both a target cost and a concatenation cost.

17. A computer program product comprising:

a computer storage device comprising at least one of a memory device or storage device;

computer program instructions stored on the computer storage device that, when processed by a computing device, instruct the computing device to perform a method for generating photo-realistic facial animation synchronized with speech, comprising;

storing, in a computer storage device, a statistical model of audiovisual data over time, based on acoustic feature vectors from actual audio data and visual feature vectors of lips images extracted from real sample images of a head and facial features of an individual during a set of utterances by the individual;

storing, in an image library, real sample images of the individual'"'"'s head and facial features during the set of utterances, the image library further storing for each of the stored real sample images the visual feature vectors obtained from the lips image extracted from the real sample image as used to generate the statistical model;

receiving an input set of acoustic feature vectors for the speech with which the facial animation is to be synchronized;

using a computer processor, applying the received input set of acoustic feature vectors to the statistical model, the statistical model thereby generating a visual feature vector sequence;

selecting, using a computer processor, a sequence of real sample images of the individual'"'"'s head and facial features from the image library, such that the selected sequence matches the visual feature vector sequence generated using the statistical model by comparing visual feature vectors in the visual feature vector sequence with visual feature vectors associated with the real sample images in the image library; and

using a computer processor, applying the selected sequence of real sample images to the three dimensional model of a head to provide the photo-realistic facial animation synchronized with the speech.

18. The computer program product of claim 17, further comprising generating the statistical model, wherein generating the statistical model comprises:

obtaining audiovisual data including a plurality of samples, including real sample images of the individual'"'"'s facial features for the set of utterances;

extracting the acoustic feature vectors and the visual feature vectors for each sample of the audiovisual data; and

training the statistical model using the acoustic feature vectors and the visual feature vectors.

19. The computer program product of claim 17, wherein selecting the sequence of real sample images comprises selecting a set of real sample images that minimizes a cost function.

20. The computer program product of claim 19, wherein the cost function comprises a target cost indicative of a difference between a visual feature vector in the generated visual feature and a visual feature vector related to a real sample image, and a concatenation cost indicative of a difference between adjacent images in the sequence of real sample images.