Facial Feature Liquifying Using Face Mesh

• US 20170132452A1
• Filed: 11/11/2015
• Published: 05/11/2017
• Est. Priority Date: 11/11/2015
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Kim, Byungmoon, Ito, Daichi, Park, Gahye
• Granted Patent: US 9,646,195 B1
• Time in Patent Office: Days
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

In embodiments of facial feature liquifying using face mesh, an image processing application is implemented to modify facial features of a face in an image using an updated face mesh generated from a combination of deformation fields. The image processing application can generate a face mesh that includes landmark points, and then construct the deformation fields on the face mesh, where the deformation fields are defined by warpable elements formed from the landmark points. The image processing application can also combine the deformation fields and generate the updated face mesh that includes the combined deformation fields. The image processing application can also display the updated face mesh and receive an input to initiate modifying one or more of the facial features of the face in the image using the combined deformation fields included in the updated face mesh.

Claims

1. An image processing application implemented to modify facial features of a face in an image using an updated face mesh generated from a combination of deformation fields, the image processing application comprising:

a face mesh generator configured to generate a face mesh for the face in the image, the face mesh including landmark points of the face;

a deformation field generator configured to construct the deformation fields on the face mesh and combine the deformation fields, the deformation fields including warpable elements formed from the landmark points;

the face mesh generator further configured to generate the updated face mesh that includes the combined deformation fields; and

a modal control interface to display the updated face mesh, the modal control interface configured to receive an input to initiate modifying one or more of the facial features of the face in the image using the combined deformation fields included in the updated face mesh.

2. The image processing application as described in claim 1, wherein the warpable elements include warpable vectors formed from the landmark points and wherein, to construct the deformation fields, the deformation field generator is configured to assign normalized coordinates to the landmark points of the face mesh and group the landmark points according to different facial feature areas.

3. The image processing application as described in claim 1, wherein the warpable elements include warpable vectors formed from the landmark points and wherein to construct the deformation fields, the face mesh generator is configured to add additional landmark points to the face mesh, the deformation fields including additional warpable vectors formed from the additional landmark points.

4. The image processing application as described in claim 3, wherein to construct the deformation fields, the deformation field generator is configured to assign normalized coordinates to the landmark points and to the additional landmark points of the face mesh, each of the assigned normalized coordinates reflecting how much a respective landmark point is to change position responsive to the input being received in the modal control interface.

5. The image processing application as described in claim 1, wherein the deformation field generator is further configured to modify one or more of the facial features of the face in the image using the combined deformation fields included in the updated face mesh in response to the input to initiate modifying the one or more of the facial features of the face in the image.

6. The image processing application as described in claim 1, wherein to combine the deformation fields, the deformation field generator is configured to apply a linear transformation algorithm to values of each of the deformation fields.

7. The image processing application as described in claim 1, wherein each of the deformation fields corresponds to a particular facial feature, and wherein the deformation field generator is further configured to modify the two or more deformation fields in response to the input to initiate modifying the one or more of the facial features of the face in the image.

8. The image processing application as described in claim 1, wherein to combine the deformation fields, the deformation field generator is configured to assign a weight to each of the deformation fields, the weight of a deformation field indicative of the usefulness of the deformation field in modifying a particular facial feature.

9. A method implemented by an image processing application for modifying facial features of a face in an image using an updated face mesh generated from a combination of deformation fields, the method comprising:

generating a face mesh for the face in the image, the face mesh including landmark points of the face;

constructing the deformation fields on the face mesh, the deformation fields including warpable elements formed from the landmark points;

combining the deformation fields;

generating the updated face mesh that includes the combined deformation fields; and

displaying the updated face mesh in a user interface that is configured to receive an input to initiate modifying one or more of the facial features of the face in the image using the combined deformation fields included in the updated face mesh.

10. The method as described in claim 9, wherein the warpable elements include warpable vectors formed from the landmark points, and further comprising adding additional landmark points to the face mesh, the deformation fields including additional warpable vectors formed from the additional landmark points, and wherein the warpable vectors and the additional warpable vectors in the combined deformation fields define how the facial features modify responsive to receiving the input to initiate the modifying of the one or more of the facial features of the face in the image.

11. The method as described in claim 9, wherein the constructing the deformation fields on the face mesh includes assigning normalized coordinates to the landmark points included in the face mesh.

12. The method as described in claim 11, further comprising applying different magnitudes of change to at least two different deformation fields using the assigned normalized coordinates responsive to receiving the input.

13. The method as described in claim 9, further comprising assigning normalized coordinates to different ones of the landmark points based on distances of each respective landmark point from two reference axes.

14. The method as described in claim 13, wherein the assigning the normalized coordinates includes assigning values to respective landmark points based on a distance between the respective landmark point and at least one additional landmark point.

15. A computing device for modifying a facial feature of a face included in an image using an updated face mesh generated from a combination of deformation fields, the computing device comprising:

a display device configured to display the image of the face;

one or more processors to implement an image processing application configured to perform operations comprising;

receiving an input to the image of the face to initiate modifying the facial feature of the face;

identifying the combination of deformation fields defined on the updated face mesh that corresponds to the facial feature;

modifying the combination of deformation fields based on warpable elements defined to the combination of deformation fields, and in accordance with a magnitude associated with the received input; and

displaying the altered facial feature on the display device responsive to the modifying the warpable elements in the combination of deformation fields, the altered facial feature being displayed with a natural and life-like appearance in relation to other facial features of the face.

16. The computing device of claim 15, wherein the warpable elements include warpable vectors formed from the landmark points and wherein the deformations to the combination of deformation fields are caused by deformations made to the landmark points forming each warpable vector based on normalized coordinates assigned to the landmark points that indicate a position of each landmark point relative to the other landmark points.

17. The computing device of claim 15, wherein the normalized coordinates assigned to the landmark points are determined from a set of reference axes defined on the updated face mesh to optimize an ideal face width to face height ratio.

18. The computing device of claim 15, wherein a magnitude of the deformations to the combination of deformation fields depends on weights assigned to areas within the combination of deformation fields.

19. The computing device of claim 15, wherein the receiving the input includes receiving selection of a slider control and wherein the magnitude associated with the slider control is determined from a change in movement between a start point and an end point of the slider control.

20. The computing device of claim 19, wherein the display device is further configured to display, on the display device of the computing device, modal controls that, responsive to selection, cause deformations to one or more of eye height, eye size, eye width, eye tilt, upper lip, lower lip, mouth width, nose width, smile, face width, jaw shape, and chin length of the face in the image.