Systems and methods for generating computer ready animation models of a human head from captured data images

• US 10,062,198 B2
• Filed: 06/23/2017
• Issued: 08/28/2018
• Est. Priority Date: 06/23/2016
• Status: Active Grant

• Current Assignee: LoomAi, Inc.
• Original Assignee: LoomAi, Inc.
• Inventors: Bhat, Kiran, Garg, Akash, Flynn, Michael Daniel, Welch, Will
• Primary Examiner(s): Chin, Michelle
• Application Number:
  US15/632,251
• Publication Number:
  US 20170372505A1
• Time in Patent Office: 431 Days
• Field of Search: None
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06T 13/205 driven by audio data
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06T 7/90 : Determination of colour characteristics
• CPC Class Codes: G06V 40/162 : using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/166 : using acquisition arrangements
• CPC Class Codes: G06V 40/171 : Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

System and methods for computer animations of 3D models of heads generated from images of faces is disclosed. A 2D captured image that includes an image of a face can be received and used to generate a static 3D model of a head. A rig can be fit to the static 3D model to generate an animation-ready 3D generative model. Sets of rigs can be parameters that each map to particular sounds. These mappings can be used to generate a playlists of sets of rig parameters based upon received audio content. The playlist may be played in synchronization with an audio rendition of the audio content.

Claims

1. A system for use in generating computer animations from captured video images comprising:

one or more processors;

memory accessible by each of the one or more processors; and

instructions stored in the memory that are read by the one or more processors and direct the one or more processors to;

receive a video segment including at least one image wherein each at least one image includes an image of a source surface;

obtain an animation rig comprising a plurality of blend shapes; and

generate animation curves that drive the animation rig to animate a 3D model from the at least one image in the video segment by;

identifying features of the source surface in the at least one image from the video segment,

mapping the identified features of the source surface to the plurality of blend shapes of the animation rig,

tracking the identified features of the source surface in the at least one images of the video segment, and

determining animation curves from the tracking of the identified features mapped to the plurality blend shapes of the animation rig.

2. The system of claim 1 wherein the mapping of the identified features to blend shapes of the animation rig is performed using a Convolutional Neural Network (CNN).

3. The system of claim 1 wherein the mapping of the identified features to the blend shapes is performed using a Recurrent Neural Network (RNN).

4. The system of claim 1 wherein the tracking of the identified features is performed using a Convolutional Neural Network (CNN).

5. The system of claim 1 wherein the tracking of the identified features is performed using a Recurrent Neural Network (RNN).

6. The system of claim 1 wherein the at least one image in the video segment is a plurality of sequential images and the animation curves for all of the plurality of sequential images are determined based on all of the plurality of sequential images.

7. The system of claim 1 wherein the at least one image in the video segment is a plurality of sequential images and the animation curves for one of the plurality of sequential images is determined based on preceding images in the plurality of sequential images.

8. The system of claim 1 wherein the instructions stored in the memory that are read by the one or more processors further include instructions for directing the one or more processors to receive depth information for the video segment and use the depth information for at least one of:

identifying of features in the at least one image;

tracking the identified features in the at least one image;

mapping the identified features to the blend shapes, and generating the animation curves.

9. The system of claim 1 wherein the instructions stored in the memory that are read by the one or more processors further include instructions for directing the one or more processors to:

receive audio data synchronized to the video segment where the audio data includes at least one sample of audio; and

map the at least one audio sample to the animations curves associated with images in the video segment synchronized to the sample of audio.

10. The system of claim 1 wherein the instructions stored in the memory that are read by the one or more processors further include instructions for directing the one or more processors to:

identify a stimuli in the at least one image of the video segment; and

map the stimuli to at least one animation curve generated from the video segment.

11. The system of claim 1 wherein the tracking of identified features in the at least one images of the video segment is performed on a per pixel basis.

12. The system of claim 1 wherein the instructions to generate the animation curves further include instructions that direct the one or more processors to:

generate video images using the generated animation curves, the animation rig and a 3D model where the rig is mapped to blend shapes of the 3D model;

compare the generated video images to animation of the video segment; and

determine whether animation of the generated video images is realistic based on the comparison.

13. The system of claim 12 wherein the instructions stored in the memory that are read by the one or more processors further include instructions that direct the one or more processors to add noise to the generated video images prior to the comparison.

14. The system of claim 12 wherein the comparing of the generated video image to the animation of the video segment is performed by a discriminator network, where the discriminator network is a neural network.

15. The system of claim 12 wherein the instructions stored in the memory that are read by the one or more processors further include instructions that direct the one or more processors to perform an optimization process on the animation of the generated video images based upon the comparison.

16. The system of claim 15 wherein the optimization process solves ambiguity problems in the animation of the generated video segments.

17. The system of claim 1 wherein the instructions stored in the memory that are read by the one or more processors further include instructions directing the one or more processors to:

obtain a 3D model based upon the source surface; and

map blend shapes of the source surface to the blend shapes of the 3D model based on the identified features of the source surface.

18. The system of claim 17 wherein the instructions stored in the memory that are read by the one or more processors further includes instructions for directing the one or more processors to generate an animated sequence of images from the generated animation curves, the animation rig, and the mapping of the blend shapes of the animation rig to the blend shape of the 3D model.

19. The system of claim 1 wherein the generating of the animation curves is performed in a plurality of stages wherein each stage determines animation curves for a different subset of features of the source surface.

20. The system of claim 19 wherein the source surface is a face and the plurality of stages includes at least one of a rigid solve stage that determines animation curves for non-deforming head features, a mouth solve stage that determines animation curves for a chin position and mouth shape networks, and an upper face solve stage that determines animation curves for facial features that move independently of the mouth shape networks.

21. A system generating computer animations from captured video images comprising:

one or more processors;

memory accessible by each of the one or more processors; and

instructions stored in the memory that are read by the one or more processors and direct the one or more processors to;

receive a video segment including at least one image wherein each at least one image includes an image of a source surface;

obtain an animation rig comprising a plurality of blends shapes;

generate animation curves that drive the animation rig to animate a 3D model from the at least one image in the video segment by;

identifying features of the source surface in the at least one image from the video segment using a deep learning neural network,

mapping the identified features of the source surface to the plurality of blend shapes of the animation rig,

tracking the identified features of the source surface in the at least one images of the video segment using a deep learning neural network, and

determining animation curves from the tracking of the identified features mapped to the plurality blend shapes of the animation rig wherein the animation curves are applied to the animation rig mapped to a 3D model to manipulate the blend shapes of the 3D model to generate computer animation;

generate video images using the generated animation curves, the animation rig and a 3D model where the rig is mapped to blend shapes of the 3D model;

add noise to the generated video images;

compare the generated video images with added noise to animation of the video segment using a discriminator neural network; and

perform an optimization process on the animation of the generated video images based upon the comparison.