Dynamic real-time generation of three-dimensional avatar models of users based on live visual input of users' appearance and computer systems and computer-implemented methods directed to thereof

• US 10,204,438 B2
• Filed: 04/18/2018
• Issued: 02/12/2019
• Est. Priority Date: 04/18/2017
• Status: Active Grant

• Current Assignee: Banuba Ltd.
• Original Assignee: Banuba Ltd.
• Inventors: Hushchyn, Yury, Kachatkou, Dzmitry
• Primary Examiner(s): Sheng, Xin
• Application Number:
  US15/956,450
• Publication Number:
  US 20180300927A1
• Time in Patent Office: 300 Days
• Field of Search: None
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/30 Polynomial surface description
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

In some embodiments, the present invention provides for a computer system that may include a camera component configured to acquire a visual content, where the visual content includes a plurality of frames having a visual representation of a person'"'"'s face; and a processor configured to: train a face detection regressor with a synthetic face model database to obtain a face detection trained regressor; apply, for each frame, the face detection trained regressor to detect or to track the face based on facial features, local features, and a pre-defined hyperparameter; construct an intermediate multi-dimensional face model; apply machine learning to determine features of an intermediate multi-dimensional head model; construct a multi-dimensional avatar; and utilize the multi-dimensional avatar to perform an activity associated with the person.

Claims

1. A method, comprising:

training, by at least one processor, at least one multi-dimensional face detection regressor with at least one synthetic face model database to obtain at least one multi-dimensional face detection trained regressor that is trained to utilize a plurality of latent variables of at least one multi-dimensional morphable model (MDMM) to predict a multi-dimensional face or a multi-dimensional head from a plurality of local binary features extracted from a visual input having a face of at least one person;

wherein the at least one synthetic face model database is distinct from the visual input having the face of the at least one person;

wherein the plurality of latent variables of the MDMM defines one or more of;

i) at least one hair-related parameter,

ii) at least one eye-related parameter,

iii) at least one skin-related parameter,

iv) at least one gender-related parameter,

v) at least one race-related parameter, and

vi) at least one age-related parameter;

obtaining, by the at least one processor, the visual input having a visual representation of the face of the at least one person;

extracting, by the at least one processor, the plurality of local binary features from the visual input;

applying, by the at least one processor, the at least one multi-dimensional face detection trained regressor to detect or to track the at least one person in the visual input by generating or updating a person-specific predicted intermediate MDMM of the face or a head of the at least one person, wherein the person-specific predicted intermediate MDMM of the face or the head of the at least one person comprises a plurality of person-specific latent variables obtained by fitting the plurality of latent variables to the visual representation of the face of the at least one person based, at least in part, on the plurality of local binary features;

wherein the plurality of person-specific latent variables of the person-specific predicted intermediate MDMM of the face or the head of the at least one person defines one or more of;

i) at least one person-specific hair-related parameter of the at least one person,

ii) at least one person-specific eye-related parameter of the at least one person,

iii) at least one person-specific skin-related parameter of the at least one person,

iv) at least one person-specific gender-related parameter of the at least one person,

v) at least one person-specific race-related parameter of the at least one person, and

vi) at least one person-specific age-related parameter of the at least one person;

constructing or updating, by the at least one processor, at least one person-specific multi-dimensional avatar of the at least one person based, at least in part, on the person-specific predicted intermediate MDMM of the face or the head of the at least one person; and

utilizing, by the at least one processor, the at least one person-specific multi-dimensional avatar of the at least one person to perform at least one activity associated with the at least one person.

2. The method of claim 1, wherein the visual input comprises a plurality of frames of a video stream.

3. The method of claim 2, wherein the video stream is a real-time video stream.

4. The method of claim 3, wherein the real-time video stream is a live video stream.

5. The method of claim 1, wherein the at least one multi-dimensional face detection regressor is a combined cascaded regressor that is based on a combination of distinct machine learning algorithms.

6. The method of claim 1, wherein the plurality of latent variables of the MDMM is selected from the group consisting of:

three-dimensional angles, translates, light vector coordinates, hair styles, beards, anthropometric parameters, and facial expressions.

7. The method of claim 1, further comprising:

transforming, by the at least one processor, the person-specific predicted intermediate MDMM of the face or the head of the at least one person to an updated person-specific predicted intermediate MDMM of the face or the head of the at least one person by applying at least one of;

averaging at least two hyperparameters;

averaging at least two hyperparameters in a principal component analysis (PCA) space;

averaging three-dimensional facial points of at least one facial landmark;

averaging two-dimensional facial landmarks; and

any combination thereof.

8. The method of claim 1, further comprising:

determining, by the at least one processor, a plurality of predicted facial landmarks of the at least one person from the plurality of person-specific latent variables.

9. The method of claim 1, wherein the visual input is obtained by a camera of a portable electronic device and wherein the at least one processor is at least one first processor of the portable electronic device.

10. The method of claim 9, wherein the training of the at least one multi-dimensional face detection regressor is performed by at least one second processor;

and wherein the at least one second processor is distinct from the at least one first processor of the portable electronic device and is remotely located from the portable electronic device.

11. A system comprising:

a camera component, wherein the camera component is configured to acquire a visual content, wherein the visual content comprises a visual representation of a face of at least one person; and

at least one processor configured to;

train at least one multi-dimensional face detection regressor with at least one synthetic face model database to obtain at least one multi-dimensional face detection trained regressor that is trained to utilize a plurality of latent variables of at least one multi-dimensional morphable model (MDMM) to predict a multi-dimensional face or a multi-dimensional head from a plurality of local binary features extracted from a visual input having a face of at least one person;

wherein the at least one synthetic face model database is distinct from the visual input having the face of the at least one person;

wherein the plurality of latent variables of the MDMM defines one or more of;

i) at least one hair-related parameter,

ii) at least one eye-related parameter,

iii) at least one skin-related parameter,

iv) at least one gender-related parameter,

v) at least one race-related parameter, and

vi) at least one age-related parameter;

extract the plurality of local binary features from the visual content;

apply, the at least one multi-dimensional face detection trained regressor to detect or to track the at least one person in the visual content by generating or updating a person-specific predicted intermediate MDMM of the face or a head of the at least one person, wherein the person-specific predicted intermediate MDMM of the face or the head of the at least one person comprises a plurality of person-specific latent variables obtained by fitting the plurality of latent variables to the visual representation of the face of the at least one person based, at least in part, on the plurality of local binary features;

wherein the plurality of person-specific latent variables of the person-specific predicted intermediate MDMM of the face or the head of the at least one person defines one or more of;

i) at least one person-specific hair-related parameter of the at least one person,

ii) at least one person-specific eye-related parameter of the at least one person,

iii) at least one person-specific skin-related parameter of the at least one person,

iv) at least one person-specific gender-related parameter of the at least one person,

v) at least one person-specific race-related parameter of the at least one person, and

vi) at least one person-specific age-related parameter of the at least one person;

construct or update at least one person-specific multi-dimensional avatar of the at least one person based, at least in part, on the person-specific predicted intermediate MDMM of the face or the head of the at least one person; and

utilize the at least one person-specific multi-dimensional avatar of the at least one person to perform at least one activity associated with the at least one person.

12. The system of claim 11, wherein the visual content comprises a plurality of frames of a video stream.

13. The system of claim 12, wherein the video stream is a real-time video stream.

14. The system of claim 13, wherein the real-time video stream is a live video stream.

15. The system of claim 11, wherein the at least one multi-dimensional face detection regressor is a combined cascaded regressor that is based on a combination of distinct machine learning algorithms.

16. The system of claim 11, wherein the plurality of latent variables of the MDMM is selected from the group consisting of:

three-dimensional angles, translates, light vector coordinates, hair styles, beards, anthropometric parameters, and facial expressions.

17. The system of claim 11, wherein the at least one processor is further configured to:

transform the person-specific predicted intermediate MDMM of the face or the head of the at least one person to an updated person-specific predicted intermediate MDMM of the face or the head of the at least one person applying at least one of;

averaging at least two hyperparameters;

averaging at least two hyperparameters in a principal component analysis (PCA) space;

averaging three-dimensional facial points of at least one facial landmark;

averaging two-dimensional facial landmarks; and

any combination thereof.

18. The system of claim 11, wherein the at least one processor is further configured to:

determine a plurality of predicted facial landmarks of the at least one person from the plurality of person-specific latent variables.

19. The system of claim 11, wherein the camera component is located within a portable electronic device and wherein the at least one processor is at least one first processor of the portable electronic device.

20. The system of claim 19, further comprising:

at least one second processor configured to train the at least one multi-dimensional face detection regressor; and

wherein the at least one second processor is distinct from the at least one first processor of the portable electronic device and is remotely located from the portable electronic device.