SYSTEM AND METHOD FOR CREATING A FULL HEAD 3D MORPHABLE MODEL

• US 20180253895A1
• Filed: 03/05/2018
• Published: 09/06/2018
• Est. Priority Date: 03/03/2017
• Status: Active Grant

• Current Assignee: Augray Pvt., Ltd.
• Original Assignee: Augray Pvt., Ltd.
• Inventors: Arumugam, Swaminathan
• Granted Patent: US 10,540,817 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

The present subject matter relates to a system and a method for generating a full head 3D morphable model. In an aspect, the system includes an identification module to identify one or more feature points as a collection of principal regions of an input 2D image. A 3D reconstruction engine generates the full head 3D morphable model by mapping feature points of the principal regions with corresponding feature points of predefined trained 3D mesh models. A rigging and animation module joins the full head 3D morphable model to a body of an animatable object to obtain a final 3D morphable model. Further, in an aspect, audio content may be added to the final 3D morphable mode to cause the final 3D morphable model to lip-sync with the audio content, and rendered in one of an AR, a MR or a VR medium.

Claims

1. A system for generating a full head three-dimensional (3D) morphable model from a two-dimensional (2D) image for use in one of an augmented reality (AR), a virtual reality (VR) and a mixed reality (MR) medium, the system comprising:

an image capturing unit to capture the 2D image of an object;

an identification module to identify feature points as a collection of principal regions on a face portion of the 2D image, wherein the identification module is configured to correct the principal regions of the 2D image by re-sizing the feature points to obtain scaled principal regions; and

a 3D reconstruction engine module to generate the full head 3D morphable model by mapping the feature points of the scaled principal regions with feature points of principal regions of predefined trained 3D mesh models stored in one or more databases to select one or more predefined 3D mesh models from the predefined trained 3D mesh models, wherein each selected 3D mesh model includes at least one principal region equivalent to a corresponding scaled principal region of the 2D image.

2. The system as claimed in claim 1, wherein the 3D reconstruction engine module selects the one or more databases from databases based on at least one of age, gender, and ethnicity of the object in the 2D image.

3. The system as claimed in claim 1, wherein the predefined trained 3D mesh models are full head 3D mesh models, and wherein each full 3D mesh model includes mesh vertices and a number of mesh vertices in each 3D mesh model is in the range from about 15000 points to 45000 points, and preferably 15000 points.

4. The system as claimed in claim 1, wherein the 3D reconstruction engine module selects the one or more 3D mesh models from the predefined trained 3D mesh models by comparing values of vertex points of the principal regions of features of the predefined trained 3D mesh models with values of vertex points of the principal regions of the corresponding features of the 2D image.

5. The system as claimed in claim 1, wherein the 3D reconstruction engine module constructs the full head 3D mesh model by combining at least one principal region of the selected 3D mesh models.

6. The system as claimed in claim 1, wherein the 3D reconstruction engine applies the texture of the input 2D image on the full head 3D morphable model.

7. The system as claimed in claim 1, wherein the resizing of the principal regions includes enhancing the length and width of the principal regions of the 2D image.

8. The system as claimed in claim 1, wherein the at least one principal region of the selected mesh models is equivalent to a corresponding scaled principal region of the 2D image if values of vertex points of the at least one principal region of the selected mesh models are close to values of vertex points of the corresponding scaled principal region of the 2D image.

9. The system as claimed in claim 1, wherein the system further comprises a rigging and animation module wherein the rigging and animation module is configured to:

join a neck zone of the full head 3D morphable model with a body of an animatable object to obtain a final 3D morphable model;

add audio content to cause the final 3D morphable model to lip-sync the audio content; and

render the final 3D morphable model in one of the AR, the MR or the VR mediums with animation and the lip-sync.

10. A method for generating a full head 3D morphable model from a 2D image for use in one of an augmented reality (AR), virtual reality (VR) and mixed reality (MR) mediums, the method comprising:

capturing the input 2D image;

identifying one or more feature points or landmark as a collection of principal regions on a face portion of the 2D image;

correcting the principal regions of the 2D image by re-sizing the feature points to obtain scaled principal regions; and

generating the full head 3D morphable model by mapping the feature points of the scaled principal regions with feature points of principal regions of predefined trained 3D mesh models stored in one or more databases to select one or more 3D mesh models from the predefined trained 3D mesh models, wherein each selected 3D mesh model includes at least one principal region equivalent to a corresponding scaled principal region of the 2D image.

11. The method as claimed in claim 10, wherein the selecting the one or more 3D mesh models from the predefined trained 3D mesh models includes comparing values of vertex points of the principal regions of features of the predefined trained 3D mesh models with values of vertex points of the principal regions of the corresponding features of the 2D image.

12. The method as claimed in claim 10, wherein the selecting the one or more databases from databases for construction of the full head 3D morphable model is based on at least one of age, gender, and ethnicity of the object.

13. The method as claimed in claim 10, wherein the predefined trained 3D mesh models are full head 3D mesh models, and wherein each full 3D mesh model includes mesh vertices and a number of mesh vertices in each 3D mesh model is in the range from about 15000 points to 45000 points, and preferably 15000 points.

14. The method as claimed in claim 10, wherein the method comprising constructing the full head 3D mesh model by combining the at least one principal region of the selected predefined trained 3D mesh models.

15. The method as claimed in claim 10, wherein the method comprising applying a texture for the full head 3D morphable model based on texture of the input 2D image.

16. The method as claimed in claim 10, wherein the method further comprises:

rigging the full head 3D morphable model to a body of an animatable object to obtain a final 3D morphable model;

animating the final 3D morphable head model by adding audio content to cause the final 3D morphable model to lip-sync the corresponding audio content; and

rendering the final 3D morphable model in one of the AR, the MR or the VR display mediums.

17. The method as claimed in claim 10, wherein the audio content is the content of the voice call that is part of the AR, the MR or the VR mediums in a smart device used for communication of voice.