Systems and methods for animating models from audio data

• US 10,169,905 B2
• Filed: 01/31/2018
• Issued: 01/01/2019
• Est. Priority Date: 06/23/2016
• Status: Active Grant

• Current Assignee: LoomAi, Inc.
• Original Assignee: LoomAi, Inc.
• Inventors: Bhat, Kiran, Garg, Akash, Flynn, Michael Daniel, Welch, Will
• Primary Examiner(s): Chin, Michelle
• Application Number:
  US15/885,667
• Publication Number:
  US 20180174348A1
• Time in Patent Office: 335 Days
• Field of Search: None
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06T 13/205 driven by audio data
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06T 7/90 : Determination of colour characteristics
• CPC Class Codes: G06V 40/162 : using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/166 : using acquisition arrangements
• CPC Class Codes: G06V 40/171 : Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

System and methods for computer animations of 3D models of heads generated from images of faces is disclosed. A 2D captured image that includes an image of a face can be received and used to generate a static 3D model of a head. A rig can be fit to the static 3D model to generate an animation-ready 3D generative model. Sets of rigs can be parameters that each map to particular sounds. These mappings can be used to generate a playlists of sets of rig parameters based upon received audio content. The playlist may be played in synchronization with an audio rendition of the audio content.

Claims

1. A system for use in generating computer animations from captured audio data comprising:

one or more processors;

memory accessible by each of the one or more processors; and

instructions stored in the memory that are read by the one or more processors and direct the one or more processors to;

receive a set of audio data representing speech of an individual;

generate animation curves that drive an animation rig to animate a 3D model based on the set of audio data by;

identifying portions of the audio data,

identifying a set of rig parameters for each identified portion of the audio data to generate a sequence of rig parameters, and

determining animation curves for animating the animation rig from the generated sequence of rig parameters.

2. The system of claim 1, wherein the set of audio data comprises text with a plurality of words, wherein each word is a portion of the audio data.

3. The system of claim 1, wherein the set of audio data comprises recorded audio content, wherein each portion is a sampled portion of the audio content.

4. The system of claim 3, wherein the received audio data further comprises synchronized video data, wherein identifying portions of the audio data comprises sampling portions of the audio content at a framerate of the synchronized video data.

5. The system of claim 1, wherein the set of audio data comprises generated audio content that is generated from text with a plurality of words.

6. The system of claim 1, wherein identifying the set of rig parameters comprises selecting a set of rig parameters that is mapped to the identified portion of the audio data from a plurality of rig parameters that are mapped to different possible portions of audio data.

7. The system of claim 6 further comprising instructions for generating the mapped plurality of rig parameters based on synchronized audio and video data.

8. The system of claim 7, wherein the synchronized audio and video data comprises recorded audio and video content of the individual.

9. The system of claim 7, wherein the synchronized audio and video data comprises recorded audio and video content of a plurality of different individuals.

10. The system of claim 7, wherein generating the mapped plurality of rig parameters comprises using a learning engine to generate mappings between each set of rig parameters and a set of samples of the input audio.

11. The system of claim 10, wherein the learning engine is a convolutional neural network.

12. The system of claim 7, wherein rig parameters of the plurality of rig parameters are nonlinearly related to shape changes for the 3D model.

13. The system of claim 7, wherein generating the mapped plurality of rig parameters further comprises:

generating video images using the generated animation curves, the animation rig and a 3D model where the animation rig is mapped to blend shapes of the 3D model;

comparing the generated video images to the synchronized video data;

determining whether animation of the generated video images is realistic based on the comparison; and

optimizing the animation of the generated video images based on the comparison.

14. The system of claim 13, wherein generating the mapped plurality of rig parameters further comprises adding noise to the generated video images prior to the comparison.

15. The system of claim 13, wherein the comparing of the generated video image to the synchronized video data is performed by a discriminator network, where the discriminator network is a neural network.

16. The system of claim 13, wherein generating the mapped plurality of rig parameters further comprises instructions for:

mapping identified features in the video data to blend shapes of the animation rig; and

tracking the identified features in the video data.

17. The system of claim 16, wherein the mapping of the identified features to blend shapes of the animation rig is performed using at least one of a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN).

18. The system of claim 16, wherein the tracking of the identified features is performed using at least one of a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN).

19. The system of claim 16, wherein the tracking of identified features in the video data is performed on a per pixel basis.

20. The system of claim 1, wherein the generating of the animation curves is performed in a plurality of stages wherein each stage determines animation curves for a different subset of features of the 3D model.

21. The system of claim 1, wherein the system further comprises instructions for directing the one or more processors to generate an animated sequence of images from the generated animation curves, the animation rig, and the mapping of the blend shapes of the animation rig to the blend shapes of the 3D model.