Facial feature liquifying using face mesh

• US 10,223,767 B2
• Filed: 05/01/2017
• Issued: 03/05/2019
• Est. Priority Date: 11/11/2015
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Kim, Byungmoon, Ito, Daichi, Park, Gahye
• Primary Examiner(s): Strege, John B
• Application Number:
  US15/583,465
• Publication Number:
  US 20170236250A1
• Time in Patent Office: 673 Days
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

In embodiments of facial feature liquifying using face mesh, an image processing application is implemented to modify facial features of a face in an image from a combination of deformation fields. The image processing application can generate a face mesh that includes landmark points, and then construct the deformation fields on the face mesh, where the deformation fields are defined by warpable elements formed from the landmark points. The image processing application can also combine the deformation fields. The image processing application can also receive an input to initiate modifying one or more of the facial features of the face in the image using the combined deformation fields.

Claims

1. An image processing application implemented to modify facial features of a face in an image from a combination of deformation fields, the image processing application comprising:

a deformation field generator configured to construct the deformation fields on a face mesh of the face in the image, and combine the deformation fields, the deformation fields including warpable elements;

a face mesh generator configured to generate an updated face mesh that includes the combined deformation fields, and one or more of the facial features of the face in the image modifiable using the combined deformation fields included in the updated face mesh; and

the deformation field generator configured to modify the one or more facial features of the face in the image using the combined deformation fields included in the updated face mesh.

2. The image processing application as described in claim 1, wherein the warpable elements include warpable vectors formed from landmark points of the face mesh and wherein, to construct the deformation fields, the deformation field generator is configured to assign normalized coordinates to the landmark points and group the landmark points according to different facial feature areas.

3. The image processing application as described in claim 1, wherein the warpable elements include warpable vectors formed from landmark points of the face mesh and wherein, to construct the deformation fields, the deformation field generator is configured to add additional landmark points to the face mesh, the deformation fields including additional warpable vectors formed from the additional landmark points.

4. The image processing application as described in claim 1, wherein to construct the deformation fields, the deformation field generator is configured to assign normalized coordinates to landmark points of the face mesh, each of the assigned normalized coordinates reflecting how much a respective landmark point is to change position responsive to an input being received to initiate modifying the one or more facial features of the face in the image.

5. The image processing application as described in claim 1, wherein the deformation field generator is said configured to modify the one or more facial features of the face in the image using the combined deformation fields included in the updated face mesh in response to an input being received to initiate modifying the one or more facial features of the face in the image.

6. The image processing application as described in claim 1, wherein to combine the deformation fields, the deformation field generator is configured to apply a linear transformation algorithm to values of each of the deformation fields.

7. The image processing application as described in claim 1, wherein each of the deformation fields corresponds to a particular facial feature.

8. The image processing application as described in claim 1, wherein to combine the deformation fields, the deformation field generator is configured to assign a weight to each of the deformation fields, the weight of a deformation field based on a usefulness of the deformation field in modifying a particular facial feature of the face.

9. A method implemented by an image processing application to modify facial features of a face in an image from a combination of deformation fields, the method comprising:

constructing the deformation fields on a face mesh of the face in the image, the deformation fields including warpable elements;

combining the deformation fields;

generating an updated face mesh that includes the combined deformation fields, whereby one or more facial features of the face in the image are modifiable using the combined deformation fields included in the updated face mesh; and

modifying the combination of deformation fields based on a received input to alter the image of the face.

10. The method as described in claim 9, wherein the warpable elements include warpable vectors formed from landmark points of the face mesh, and wherein the warpable vectors in the combined deformation fields define how the one or more facial features are modifiable.

11. The method as described in claim 9, wherein said constructing the deformation fields on the face mesh includes assigning normalized coordinates to landmark points included in the face mesh.

12. The method as described in claim 9, further comprising:

receiving the input to the updated face mesh to initiate the modifying a facial feature of the face in the image;

said modifying the combination of deformation fields in accordance with a magnitude associated with the received input; and

altering the image of the face based on said modifying the combination of deformation fields.

13. The method as described in claim 9, wherein the warpable elements are formed from landmark points of the face mesh, and further comprising assigning normalized coordinates to the landmark points based on distances of each respective landmark point from two reference axes and/or from at least one additional landmark point.

14. The method as described in claim 9, wherein the warpable elements include warpable vectors formed from landmark points of the face mesh;

and the method further comprising;

assigning normalized coordinates to the landmark points and grouping the landmark points according to different facial feature areas of the face mesh.

15. A computing device comprising:

a display device configured to display an image of a face that includes modifiable facial features;

one or more processors to implement an image processing application configured to perform operations comprising;

receiving an input to the image of the face to initiate modifying a facial feature of the face;

identifying a combination of deformation fields defined on a face mesh, the combination of deformation fields corresponding to the modifiable facial feature and including warpable elements;

modifying the combination of deformation fields in accordance with a magnitude associated with the received input; and

altering the facial feature on the display device based on said modifying the combination of deformation fields corresponding to the facial feature.

16. The computing device of claim 15, wherein the warpable elements include warpable vectors formed from landmark points of the face mesh and wherein said modifying the combination of deformation fields is caused by deformations made to the landmark points forming each warpable vector.

17. The computing device of claim 15, wherein normalized coordinates are assigned to landmark points of the face mesh, the normalized coordinates determined from a set of reference axes defined on the face mesh to optimize an ideal face width to face height ratio.

18. The computing device of claim 15, wherein a magnitude of said modifying the combination of deformation fields depends on weights assigned to areas within the combination of deformation fields.

19. The computing device of claim 15, wherein said receiving the input includes receiving selection of a slider control, and wherein a magnitude associated with the selection of the slider control is determined from a change in movement between a start point and an end point of the slider control.

20. The computing device of claim 19, wherein the display device is further configured to display modal controls that, responsive to selection, cause deformations to one or more of eye height, eye size, eye width, eye tilt, upper lip, lower lip, mouth width, nose width, smile, face width, jaw shape, and chin length of the face in the image.