Systems and methods for generating computer ready animation models of a human head from captured data images

• US 10,559,111 B2
• Filed: 12/14/2018
• Issued: 02/11/2020
• Est. Priority Date: 06/23/2016
• Status: Active Grant

• Current Assignee: LoomAi, Inc.
• Original Assignee: LoomAi, Inc.
• Inventors: Sachs, Ian, Bhat, Kiran, Monn, Dominic, Radhakrishnan, Senthil, Welch, Will
• Primary Examiner(s): Chin, Michelle
• Application Number:
  US16/221,321
• Publication Number:
  US 20190122411A1
• Time in Patent Office: 424 Days
• Field of Search: None
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06T 13/205 driven by audio data
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 : involving reference images or patches
• CPC Class Codes: G06T 7/55 : from multiple images
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06T 7/90 : Determination of colour characteristics
• CPC Class Codes: G06V 10/454 : Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 : using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 : using neural networks
• CPC Class Codes: G06V 20/647 : by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/162 : using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/166 : using acquisition arrangements
• CPC Class Codes: G06V 40/171 : Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/175 : Static expression
• CPC Class Codes: H04N 7/157 : defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

System and methods for computer animations of 3D models of heads generated from images of faces is disclosed. A 2D captured image that includes an image of a face can be received and used to generate a static 3D model of a head. A rig can be fit to the static 3D model to generate an animation-ready 3D generative model. Sets of rigs can be parameters that each map to particular sounds or particular facial movement observed in a video. These mappings can be used to generate a playlists of sets of rig parameters based upon received audio or video content. The playlist may be played in synchronization with an audio rendition of the audio content. Methods can receive a captured image, identify taxonomy attributes from the captured image, select a template model for the captured image, and perform a shape solve for the selected template model based on the identified taxonomy attributes.

Claims

1. A method for generating a three dimensional (3D) head model from a captured image, the method comprising:

receiving a captured image;

identifying a set of taxonomy attributes from the captured image by;

using a single encoder model to generate an embedding of the captured image;

using a set of one or more terminal models to analyze the generated embedding; and

identifying taxonomy attributes based on the classifications of the terminal models;

selecting a template model for the captured image; and

performing a shape solve for the selected template model based on the identified taxonomy attributes.

2. The method of claim 1, wherein the set of taxonomy attributes comprises a set of one or more local taxonomy attributes and a set of one or more global taxonomy attributes.

3. The method of claim 2 further comprising performing a texture synthesis based on one or more of the taxonomy attributes.

4. The method of claim 2, wherein the template model is selected based on the set of global taxonomy attributes.

5. The method of claim 2, wherein identifying the set of global taxonomy attributes comprises using a classifier model to classify the captured image.

6. The method of claim 2, wherein the set of global taxonomy attributes comprises at least one of an ethnicity and a gender.

7. The method of claim 2, wherein the set of local taxonomy attributes comprise one or more of head shape, eyelid turn, eyelid height, nose width, nose turn, mustache, beard, sideburns, eye rotation, jaw angle, lip thickness, and chin shape.

8. The method of claim 1, wherein each terminal model of the set of terminal models produces a score for an associated local taxonomy attribute, wherein performing the shape solve for a given local taxonomy attribute is based on the score calculated for the local taxonomy attribute.

9. The method of claim 1 further comprising:

training the single encoder model and the set of terminal models, wherein each terminal model is trained to classify a different taxonomy attribute.

10. The method of claim 9, wherein training the encoder comprises:

generating an embedding from the single encoder model;

calculating a first loss for a first terminal model based on the generated embedding;

back propagating the first calculated loss through the encoder;

calculating a second loss for a second terminal model based on the generated embedding; and

back propagating the second calculated loss through the single encoder model.

11. The method of claim 1, wherein receiving the captured image comprises capturing an image with a camera of a mobile device, wherein performing the shape solve is performed on the same mobile device.

12. A non-transitory machine readable medium containing processor instructions for generating a three dimensional (3D) head model from a captured image, where execution of the instructions by a processor causes the processor to perform a process that comprises:

receiving a captured image;

identifying a set of taxonomy attributes from the captured image by;

using a single encoder model to generate an embedding of the captured image;

using a set of one or more terminal models to analyze the generated embedding; and

identifying taxonomy attributes based on the classifications of the terminal models;

selecting a template model for the captured image; and

performing a shape solve for the selected template model based on the identified taxonomy attributes.

13. The non-transitory machine readable medium of claim 12, wherein the set of taxonomy attributes comprises a set of one or more local taxonomy attributes and a set of one or more global taxonomy attributes, wherein the template model is selected based on the set of global attributes.

14. The non-transitory machine readable medium of claim 13, wherein the set of global taxonomy attributes comprises at least one of an ethnicity and a gender.

15. The non-transitory machine readable medium of claim 13, wherein execution of the instruction further causes the processor to perform a texture synthesis based on one or more of the taxonomy attributes.

16. The non-transitory machine readable medium of claim 12, wherein the process further comprises training the single encoder model and the set of terminal models, wherein each terminal model is trained to classify a different taxonomy attribute.

17. The non-transitory machine readable medium of claim 16, wherein training the encoder comprises:

generating an embedding from the single encoder model;

calculating a first loss for a first terminal model based on the generated embedding;

back propagating the first calculated loss through the encoder;

calculating a second loss for a second terminal model based on the generated embedding; and

back propagating the second calculated loss through the single encoder model.

18. The non-transitory machine readable medium of claim 12, wherein receiving the captured image comprises capturing an image with a camera of a mobile device, wherein performing the shape solve is performed on the same mobile device.