Automated avatar generation

• US 11,048,916 B2
• Filed: 05/10/2019
• Issued: 06/29/2021
• Est. Priority Date: 03/31/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Gusarov, Maksim, Kudriashov, Igor, Filev, Valerii, Kotcur, Sergei
• Primary Examiner(s): Tucker, Wesley J
• Application Number:
  US16/409,390
• Publication Number:
  US 20190266390A1
• Time in Patent Office: 781 Days
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

Systems, devices, media, and methods are presented for generating facial representations using image segmentation with a client device. The systems and methods receive an image depicting a face, detect at least a portion of the face within the image, and identify a set of facial landmarks within the portion of the face. The systems and methods determine one or more characteristics representing the portion of the face, in response to detecting the portion of the face. Based on the one or more characteristics and the set of facial landmarks, the systems and methods generate a representation of a face.

Claims

1. A method, comprising:

receiving, by one or more processors, a two-dimensional (2D) image depicting at least a portion of a face of a first user;

determining, by the one or more processors, a hair texture of a hair region of the face of the first user depicted in the 2D image;

comparing a dimension of the hair region to one or more of a set of facial landmarks within a portion of the face; and

determining hair length based on the comparison of the dimension of the hair region to the one or more of the set of facial landmarks;

generating, by the one or more processors, a representation of the face of the first user depicted in the 2D image based on the determined hair texture and hair length of the hair region;

displaying a user interface element associated with a physical attribute of the generated representation of the face;

in response to receiving input selecting the user interface element associated with the physical attribute of the generated representation of the face, presenting a plurality of user interface elements for modifying the physical attribute of the generated representation of the face, each of the user interface elements in the plurality of user interface elements representing a different modification to a same physical attribute of the generated representation of the face; and

generating, by a user device of the first user, a message, directed to a second user, that includes the generated representation of the face of the first user.

2. The method of claim 1 further comprising:

presenting a confirmation message that includes the representation of the face and an acceptance user interface element and a rejection user interface element;

in response to receiving input selecting the rejection user interface element, causing presentation of the plurality of user interface elements for modifying the physical attribute of the generated representation of the face; and

in response to receiving input selecting one of the plurality of user interface elements, modifying the representation of the face based on the selected one of the plurality of user interface elements.

3. The method of claim 1 further comprising:

presenting a message associated with the physical attribute that includes the representation of the face and an acceptance user interface element and a rejection user interface element;

in response to receiving input selecting the rejection user interface element, causing presentation of the plurality of user interface elements for modifying a first type of physical attribute of the generated representation of the face; and

in response to receiving input selecting one of the plurality of user interface elements, modifying the representation of the face based on the selected one of the plurality of user interface elements.

4. The method of claim 1, wherein determining the hair texture comprises:

identifying a set of colors within the hair region of the face; and

determining variations in the set of colors that indicate edges, objects or shapes within hair associated with the hair texture.

5. The method of claim 4 further comprising:

comparing a dimension of the hair region to the one or more of a set of facial landmarks; and

determining a hair length based on the comparison of the dimension of the hair region to the one or more of the set of facial landmarks.

6. The method of claim 1 further comprising selecting a set of hat templates from clothing templates in response to determining that an obstruction obscures the hair region of a portion of facial landmarks representing a forehead of the face.

7. The method of claim 1, further comprising:

identifying the set of facial landmarks within the portion of the face; and

in response to identifying the set of facial landmarks, determining one or more characteristics representing the portion of the face depicted in the 2D image by;

identifying a set of colors within the hair region of the face, wherein the determination of the hair texture is based on variations in the set of colors that indicate edges, objects or shapes within hair associated with the hair texture.

8. The method of claim 1 further comprising activating a given user interface element of the plurality of user interface elements to perform the modification associated with the given user interface element in response to detecting touch at an area of a display occupied by the given user interface element.

9. The method of claim 1 further comprising:

determining a first physical attribute type of the portion of the face based on one or more distances between two or more facial landmarks; and

determining a second physical attribute type by identifying an area of interest on the portion of the face and extracting an average color depicted within the area of interest.

10. The method of claim 7, further comprising:

determining one or more distances between two or more facial landmarks of the set of facial landmarks to determine the one or more characteristics, wherein determining the distances comprises;

determining a first distance between facial landmarks of the set of facial landmarks representing eyes depicted on the portion of the face;

determining a second distance between facial landmarks of the set of facial landmarks representing the eyes and a nose depicted on the portion of the face;

determining a third distance between facial landmarks of the set of facial landmarks representing the eyes and a mouth depicted on the portion of the face; and

determining a fourth distance between facial landmarks of the set of facial landmarks representing the eyes and a chin depicted on the portion of the face.

11. The method of claim 1, further comprising:

identifying one or more iris within the face; and

determining a prevailing color for the one or more iris, wherein identifying the one or more iris further comprises;

determining a shape of one or more eyes surrounding the one or more iris;

determining a height of the shape; and

determining an iris dimension based on the height of the shape of the one or more eyes.

12. The method of claim 1 further comprising:

determining one or more eyebrow regions of the face; and

identifying one or more shapes of the one or more eyebrow region.

13. The method of claim 1 further comprising:

identifying an obstruction on the face depicted in the 2D image; and

selecting a type of physical element to add to the generated representation of the face based on a location or position of the obstruction on the face.

14. The method of claim 1, wherein a first hair attribute of a plurality of hair attributes of the hair region comprises hair volume, further comprising determining the hair volume is determined by:

identifying a first portion of the set of facial landmarks that represents an outline of the face; and

computing a distance a hair region extends from the identified portion of the set of facial landmarks that represents the outline of the face to an outer opposing edge of the hair region.

15. The method of claim 14, wherein a second hair attribute of the plurality of hair attributes comprises hair length, further comprising determining the hair length by:

identifying a second portion of the set of facial landmarks that represents a chin; and

computing a distance from the hair region to the second portion of the set of facial landmarks that represents the chin.

16. The method of claim 1 further comprising:

selecting a region of the face where obstructions commonly appear;

determining whether an obstruction is present in the selected region using one or more of a set of facial landmarks; and

in response to determining that the obstruction is present, matching the obstruction to a template selected from a set of templates based on a location of the obstruction.

18. The method of claim 16, further comprising selecting a set of hat templates from clothing templates in response to determining that an obstruction obscures the hair region of a portion of facial landmarks representing a forehead of the face.

19. The method of claim 16, further comprising:

selecting a set of glasses templates in response to determining that an obstruction encompasses or is positioned proximate to facial landmarks representing eyes and nose of the face; and

selecting a set of facial hair templates in response to determining that the obstruction is positioned proximate to or obscures facial landmarks representing a mouth or jaw line of the face.

17. A system, comprising:

one or more processors; and

a non-transitory processor-readable storage medium coupled to the one or more processors and storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

receiving a two-dimensional (2D) image depicting at least a portion of a face of a first user;

determining a hair texture of a hair region of the face of the first user depicted in the 2D image;

comparing a dimension of the hair region to one or more of a set of facial landmarks within a portion of the face; and

determining hair length based on the comparison of the dimension of the hair region to the one or more of the set of facial landmarks;

generating a representation of the face of the first user depicted in the 2D image based on the determined hair texture and hair length of the hair region;

displaying a user interface element associated with a physical attribute of the generated representation of the face;

in response to receiving input selecting the user interface element associated with the physical attribute of the generated representation of the face, presenting a plurality of user interface elements for modifying the physical attribute of the generated representation of the face, each of the user interface elements in the plurality of user interface elements represent a different modification to a same physical attribute of the generated representation of the face; and

generating, by a user device of the first user, a message, directed to a second user, that includes the generated representation of the face of the first user.

20. A non-transitory processor-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

receiving a two-dimensional (2D) image depicting at least a portion of a face of a first user;

determining a hair texture of a hair region of the face of the first user depicted in the 2D image;

comparing a dimension of the hair region to one or more of a set of facial landmarks within a portion of the face; and

determining hair length based on the comparison of the dimension of the hair region to the one or more of the set of facial landmarks;

generating a representation of the face of the first user depicted in the 2D image based on the determined hair texture and hair length of the hair region;

displaying a user interface element associated with a physical attribute of the generated representation of the face;

in response to receiving input selecting the user interface element associated with the physical attribute of the generated representation of the face, presenting a plurality of user interface elements for modifying the physical attribute of the generated representation of the face, each of the user interface elements in the plurality of user interface elements represent a different modification to a same physical attribute of the generated representation of the face; and

generating, by a user device of the first user, a message, directed to a second user, that includes the generated representation of the face of the first user.