Systems and methods for generating computer ready animation models of a human head from captured data images

• US 9,786,084 B1
• Filed: 06/27/2017
• Issued: 10/10/2017
• Est. Priority Date: 06/23/2016
• Status: Active Grant

• Current Assignee: LoomAi, Inc.
• Original Assignee: LoomAi, Inc.
• Inventors: Bhat, Kiran, Garg, Akash, Flynn, Michael Daniel, Welch, Will
• Primary Examiner(s): Chin, Michelle
• Application Number:
  US15/634,741
• Time in Patent Office: 105 Days
• Field of Search: None
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06T 13/205 driven by audio data
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06T 7/90 : Determination of colour characteristics
• CPC Class Codes: G06V 40/162 : using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/166 : using acquisition arrangements
• CPC Class Codes: G06V 40/171 : Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

System and methods for computer animations of 3D models of heads generated from images of faces is disclosed. A 2D captured image that includes an image of a face can be received and used to generate a static 3D model of a head. A rig can be fit to the static 3D model to generate an animation-ready 3D generative model. Sets of rigs can be parameters that each map to particular sounds. These mappings can be used to generate a playlists of sets of rig parameters based upon received audio content. The playlist may be played in synchronization with an audio rendition of the audio content.

Claims

1. A system for generating a three dimensional (3D) head model for use in computer animation from a captured image in which a face is visible comprising:

one or more processors;

memory readable by the one or more processors; and

instructions stored in the memory that when read by the one or more processors directs the one more processors to;

receive a captured image in which a face is visible, and

generate a customized static 3D model of a head from the captured image by;

determining a geometry component of the face visible in the captured image;

determining texture components of the face visible in the captured image;

determining lighting components of the face visible in the captured image;

determining camera properties of the captured image;

applying the geometry component, texture components, lighting components, and camera properties to a generative model to generate a customized static 3D model of a head based on the image; and

optimizing the customized static 3D model using a gradient-based optimization framework.

2. The system of claim 1 wherein the instructions for optimizing further include instructions that direct the one or more processors to:

enforcing smoothness between neighboring vertices of customized 3D model during the optimizing of the customized static 3D model using a gradient-based optimization framework by regularization of terms.

3. The system of claim 2 wherein a texture regularization term imposes a penalty for vertex color difference between neighboring vertices on a mesh of the customized 3D model and a shape regularization term imposes an edge-smoothness penalty for a deviation for undeformed edge lengths of the mesh.

4. The system of claim 1 wherein the instructions further includes instructions that direct the one or more processors to:

receive an additional captured image in which the face is visible; and

optimize the customized static 3D model based upon the additional captured image.

5. The system of claim 1 wherein the instructions further include instructions that direct the one or more processors to:

perform a segmentation process to identify a hair region of the face in the captured image.

6. The system of claim 5 wherein the instructions to perform the segmentation process includes instructions that direct the one or more processing systems to:

project the captured image onto a determined geometry of the face to generate a 3D static model of the face,

identify pixels of the face belonging to textured regions,

cluster pixels of the face by projected color,

identify regions of pixels of the face that are skin,

determine a hairline on the face from the clustered pixels, and

construct a facial boundary for the face based on the hairline.

7. The system of claim 6 wherein the instructions to perform the segmentation process further include instructions that direct the one or more processors to:

construct a hairline halo by expanding outward from the facial boundary; and

determine regions that image hair and that do not image hair in the hairline halo.

8. The system of claim 7 wherein the hairline halo is constructed by performing a morphological dilation of a skin region of the face and differencing a dilated skin region with a non-skin region of the face.

9. The system of claim 7 wherein the hairline halo is constructed by performing an erosion of a non-skin region of the face and differencing an eroded non-skin region with a skin region of the face.

10. The system of claim 1 wherein the instructions further include instructions that direct the one or more processors to fill empty portions of the customized static 3D model.

11. The system of claim 10 wherein the instructions to fill empty portions of the customized static 3D model include instructions that direct the one or more processors to:

generate a visibility mask indicating portions of the face represented in the captured image from the captured image and the determined geometry of the face,

calculate a boundary of the visible region,

smooth a boundary curve of the visibility region,

generate a new visibility mask,

identify each non-visible region of the face based upon the new visibility mask,

determine a matching visible region of each of the identified non-visible regions,

extrapolate a skin color for each visible region,

determine an extrapolated skin color for each matching visible region, and

fill-in each non-visible region with the extrapolated skin color determined for a corresponding matching visible region.

12. The system of claim 11 wherein the instructions to fill empty portions of the customized static 3D model include instructions that direct the one or more processors systems to:

add high frequency details to each of the filled-in non-visible regions of the image.

13. The system of claim 12 wherein the high frequency details are added using a Laplacian pyramid of a template texture.

14. The system of claim 1 wherein the instruction further include instructions for directing the one or more processors to:

determine a position for each of a plurality of facial landmarks in the image by performing a Mnemonic Descent Method (MDM) using a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN) that are jointly trained.

15. The system of claim 14 wherein the determining of the position of each of a plurality of landmarks is performed by:

aligning each of the landmarks at positions aligned to a center of the face in the image; and

iteratively re-calculating the position of each of the plurality of landmarks until a threshold value is met by;

obtaining a patch of pixels of a predetermined size surrounding the position of each of the plurality of landmarks,

applying the patch for each of the plurality of descriptors to the CNN to generate an N length descriptor describing each patch,

concatenating the N length descriptors of each patch to generate a descriptor encapsulating all of the patches,

projecting the descriptor encapsulating all of the patches through the RNN to determine an adjustment amount for each of the plurality of landmarks, and

updating the landmarks based on the current position of each of the plurality of landmarks and the adjustment amount of each of the plurality of landmarks.

16. The system of claim 15 wherein the CNN includes a global average pooling after a final convolution layer of the CNN to obtain a fixed output size that is invariant with size of an input patch.

17. The system of claim 16 wherein the CNN includes an additional convolutional layer that is not included in the global average pooling to retain spatial information.

18. The system of claim 1 wherein the instructions includes instructions that direct the one or more processors to:

generate a customized rig for the customized static 3D model from the captured image.

19. The system of claim 1 wherein the instructions includes instructions that direct the one or more processors to:

receive video images that include the face,

fit a model rig to blend shapes of the customized static 3D model,

track a 3D surface of the face in the received videos, and

re-compute the blend shapes of the face to best fit the tracked 3D surface of the face from the video images.

20. The system of claim 19 wherein the tracking of the 3D surfaced is performed on a per pixel basis.

21. The system of claim 19 wherein the instructions further include instructions for directing the one or more processors to:

receive video images synchronized to audio data,

track a 3D surface of the face in the frames of the video images to generate a set of rig parameters for a portion of the video images, and

map the set of rig parameters for each portion of the video images to a corresponding synchronized portion of the audio data.

22. The system of claim 21 wherein tracking of the 3D surface to at least one of generating the set of rig parameters and mapping to the audio data is performed using a temporal model including recurrent neutral network.

23. The system of claim 21 wherein the tracking of the 3D surface to generate the set of rig parameters and the mapping to the audio data is performed using a time series model including a convolutional neural network.

24. The system of claim 21 wherein the instruction further include instructions that direct the one or more processors to:

receive an input of audio data; and

generate an animation playlist from input audio data from the mapping of the sets rig parameters to corresponding synchronized portions of the audio data.

25. The system of claim 1 wherein a generative adversarial network is for at least one of mapping the set of rig parameters for each portion of the video images to a corresponding synchronized portion of the audio data and generating an animation playlist from input audio data from the mapping of the sets rig parameters to corresponding synchronized portions of the audio data.