Computing system for expressive three-dimensional facial animation

• US 11,238,885 B2
• Filed: 10/29/2018
• Issued: 02/01/2022
• Est. Priority Date: 10/29/2018
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Mittal, Gaurav, Wang, Baoyuan
• Primary Examiner(s): Colucci, Michael
• Application Number:
  US16/173,491
• Publication Number:
  US 20200135226A1
• Time in Patent Office: 1,191 Days
• Field of Search: 704235
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads
• CPC Class Codes: G10L 21/10 Transforming into visible information
• CPC Class Codes: H04L 51/10 Multimedia information

Abstract

A computer-implemented technique for animating a visual representation of a face based on spoken words of a speaker is described herein. A computing device receives an audio sequence comprising content features reflective of spoken words uttered by a speaker. The computing device generates latent content variables and latent style variables based upon the audio sequence. The latent content variables are used to synchronized movement of lips on the visual representation to the spoken words uttered by the speaker. The latent style variables are derived from an expected appearance of facial features of the speaker as the speaker utters the spoken words and are used to synchronize movement of full facial features of the visual representation to the spoken words uttered by the speaker. The computing device causes the visual representation of the face to be animated on a display based upon the latent content variables and the latent style variables.

Claims

1. A computing device, comprising:

a processor;

memory storing instructions, wherein the instructions, when executed by the processor, cause the processor to perform acts comprising;

receiving an audio sequence reflective of words uttered by a speaker;

based upon the audio sequence, generating a first set of coefficients that are indicative of lips of the speaker as the speaker utters the words, wherein the first set of coefficients are generated based upon latent content variables that have been generated via a computer-implemented model, wherein the computer-implemented model has been trained without utilization of motion capture data, wherein the latent content variables are generated by the computer-implemented model without utilization of motion capture techniques;

based upon the audio sequence, generating a second set of coefficients that are indicative of facial features of the speaker other than the lips of the speaker as the speaker utters the words, wherein the second set of coefficients are generated based upon latent style variables that have been generated via the computer-implemented model, wherein the latent style variables comprise latent identity variables that are based upon identity factors of a plurality of speaker as the plurality of speakers speak and latent emotional variables that are based upon emotions of the plurality of speakers as the plurality of speakers speak, wherein the latent style variables are generated by the computer-implemented model without utilization of the motion capture techniques;

generating a third set of coefficients based upon the first set of coefficients and the second set of coefficients; and

causing a visual representation of a face to be animated on a display based upon the third set of coefficients such that movement of lips of the visual representation reflects the words uttered by the speaker while the visual representation is animated, wherein facial features of the visual representation of the face other than the lips are synced to the lips of the visual representation, and further wherein the visual representation of the face reflects an identity of the speaker and an emotion of the speaker as the speaker utters the words while the visual representation of the face is animated.

2. The computing device of claim 1, wherein the computing device is a gaming console.

3. The computing device of claim 1, wherein the computing device receives the audio sequence by way of a microphone.

4. The computing device of claim 1, wherein the latent identity variables are influenced by at least one of an age of the speaker, a gender of the speaker, or an ethnicity of the speaker.

5. The computing device of claim 1, wherein the computing device is in network communication with a second computing device, wherein the second computing device comprises a microphone and the display, wherein the second computing device receives the audio sequence from the speaker by way of the microphone, wherein the second computing device transmits the audio sequence to the computing device, wherein the computing device transmits data to the second computing device causing the second computing device to animate the visual representation of the face on the display.

6. The computing device of claim 1, the acts further comprising:

segmenting the visual representation of the face into segments, wherein each segment in the segments is assigned to a different facial feature of the visual representation of the face, wherein causing the visual representation of the face to be animated on the display is further based upon the segments.

7. The computing device of claim 6, wherein the segments comprise:

a first segment assigned to eyes of the visual representation;

a second segment assigned to a nose of the visual representation;

a third segment assigned to a mouth of the visual representation; and

a fourth segment assigned to features of the visual representation other than the eyes, the nose, and the mouth.

8. The computing device of claim 1, wherein the computer-implemented model includes a recurrent neural network (RNN) comprising a plurality of long short-term memory (LSTM)units.

9. The computing device of claim 1, wherein the visual representation of the face is a depiction of a face of the speaker.

10. The computing device of claim 1, wherein causing the visual representation of the face to be animated on the display based upon the third set of coefficients comprises:

generating a plurality of frames based upon the third set of coefficients, wherein each frame depicts a state of the visual representation of the face at a sequential point in time; and

causing each frame in the plurality of frames to be presented sequentially on the display.

11. The computing device of claim 1, the acts further comprising:

causing the audio sequence to be played on an audio speaker concurrently with causing the visual representation of the face to be animated such that movements of the visual representation of the face are synchronized with the words of the audio sequence.

12. A method executed by a processor of a computing device, the method comprising:

receiving an audio sequence reflective of words uttered by a speaker;

based upon the audio sequence, generating a first set of coefficients that are indicative of lips of the speaker as the speaker utters the words, wherein the first set of coefficients are generated based upon latent content variables that have been generated via a computer-implemented model, wherein the computer-implemented model has been trained without utilization of motion capture data, wherein the latent content variables are generated by the computer-implemented model without utilization of motion capture techniques;

based upon the audio sequence, generating a second set of coefficients that are indicative of facial features of the speaker other than the lips of the speaker as the speaker utters the words, wherein the second set of coefficients are generated based upon latent style variables that have been generated via the computer-implemented model, wherein the latent style variables comprise latent identity variables that are based upon identity factors of a plurality of speakers as the plurality of speakers speak and latent emotional variables that are based upon emotions of the plurality of speakers as the plurality of speakers speak, wherein the latent style variables are generated by the computer-implemented model without utilization of the motion capture techniques;

generating a third set of coefficients based upon the first set of coefficients and the second set of coefficients; and

causing a visual representation of a face to be animated on a display based upon the third set of coefficients such that movement of lips of the visual representation reflects the words uttered by the speaker while the visual representation is animated, wherein facial features of the visual representation of the face other than the lips are synced to the lips of the visual representation, and further wherein the visual representation of the face reflects an identity of the speaker and an emotion of the speaker as the speaker utters the words while the visual representation of the face is animated.

13. The method of claim 12, wherein the visual representation of the face is an avatar of the speaker.

14. The method of claim 12, wherein the latent identity variables are influenced by at least one of an age of the speaker, a gender of the speaker, or an ethnicity of the speaker.

15. The method of claim 12, further comprising:

segmenting the visual representation of the face into segments, wherein each segment in the segments is assigned to a different facial feature of the visual representation of the face, wherein causing the visual representation of the face to be animated on the display is further based upon the segments, wherein the segments comprise;

a first segment assigned to eyes of the visual representation;

a second segment assigned to a nose of the visual representation;

a third segment assigned to a mouth of the visual representation; and

a fourth segment assigned to features of the visual representation other than the eyes, the nose, and the mouth.

16. The method of claim 12 further comprising:

causing the audio sequence to be played on a speaker concurrently with causing the visual representation of the face to be animated such that movements of the visual representation of the face are synchronized with the words of the audio sequence.

17. A computer-readable storage medium comprising instructions that, when executed one or more processors of a computing device, perform acts comprising:

receiving an audio sequence reflective of words uttered by a speaker by way of a microphone;

based upon the audio sequence, generating a first set of coefficients that are indicative of lips of the speaker as the speaker utters the words, wherein the first set of coefficients are generated based upon latent content variables that have been generated via a computer-implemented model, wherein the computer-implemented model has been trained without utilization of motion capture data, wherein the latent content variables are generated by the computer-implemented model without utilization of motion capture techniques;

based upon the audio sequence, generating a second set of coefficients that are indicative of facial features of the speaker other than the tips of the speaker as the speaker utters the words, wherein the second set of coefficients are generated based upon latent style variables that have been generated via the computer-implemented model, wherein the latent style variables comprise latent identity variables that are based upon identity factors of a plurality of speakers as the plurality of speakers speak and latent emotional variables that are based upon emotions of the plurality of speakers as the plurality of speakers speak, wherein the latent style variables are generated by the computer-implemented model without utilization of the motion capture techniques;

generating a third set of coefficients based upon the first set of coefficients and the second set of coefficients;

causing a visual representation of a face to be animated on a display of a second computing device that is in network communication with the computing device based upon the third set of coefficients such that movement of lips of the visual representation reflects the words uttered by the speaker while the visual representation is animated, wherein facial features of the visual representation of the face other than the lips are synced to the lips of the visual representation, and further wherein the visual representation of the face reflects an identity of the speaker and an emotion of the speaker as the sneaker utters the words while the visual representation of the face is animated; and

causing the audio sequence to be played on a speaker of the second computing device concurrently with causing the visual representation of the face to be animated such that movements of the visual representation of the face are synchronized with the words of the audio sequence.

18. The computer-readable storage medium of claim 17, wherein the microphone is comprised by a third computing device operated by the speaker, wherein the third computing device is in network communication with the computing device, wherein the audio sequence is received from the third computing device.

19. The computer-readable storage medium of claim 17, wherein the latent style variables are influenced by at least one of an age of the speaker, a gender of the speaker, or an ethnicity of the speaker.

20. The computer-readable storage medium of claim 17, wherein the one or more processors are graphics processing units (GPUs).