Automated avatar generation

• US 10,339,365 B2
• Filed: 03/31/2016
• Issued: 07/02/2019
• Est. Priority Date: 03/31/2016
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Gusarov, Maksim, Kudriashov, Igor, Filev, Valerii, Kotcur, Sergei
• Primary Examiner(s): Tucker, Wesley J
• Application Number:
  US15/086,749
• Publication Number:
  US 20170286752A1
• Time in Patent Office: 1,188 Days
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

Systems, devices, media, and methods are presented for generating facial representations using image segmentation with a client device. The systems and methods receive an image depicting a face, detect at least a portion of the face within the image, and identify a set of facial landmarks within the portion of the face. The systems and methods determine one or more characteristics representing the portion of the face, in response to detecting the portion of the face. Based on the one or more characteristics and the set of facial landmarks, the systems and methods generate a representation of a face.

Claims

1. A method, comprising:

receiving, by one or more processors, one or more two-dimensional images depicting at least a portion of a face of a first user;

detecting, by the one or more processors, the portion of the face depicted within the one or more two-dimensional images;

identifying a set of facial landmarks within the portion of the face depicted within the one or more two-dimensional images;

in response to detecting the portion of the face and the set of facial landmarks, determining one or more characteristics representing the portion of the face depicted in the one or more two-dimensional images, the determining comprising;

identifying a set of colors within a hair region of the face;

determining a hair texture for the hair region based on variations in the set of colors that indicate edges, objects or shapes within the hair associated with the hair texture;

comparing a dimension of the hair region to one or more of the set of facial landmarks; and

determining hair length or hair volume based on the comparison of the dimension of the hair region to the one or more of the set of facial landmarks;

based on the one or more characteristics and the set of facial landmarks, generating a representation of a face for the at least one portion of the face depicted in the one or more images;

generating one or more sticker graphics incorporating the representation of the face, the generating of the one or more sticker graphics comprising generating a scaled version of the representation corresponding to a template graphic; and

generating, by a user device of the first user, a message, directed to a second user, that includes the one or more sticker graphics incorporating the representation of the face of the first user.

2. The method of claim 1, wherein identifying the set of facial landmarks further comprises:

determining one or more distances between two or more facial landmarks of the set of facial landmarks.

3. The method of claim 2, wherein determining the distances further comprises:

determining a first distance between facial landmarks of the set of facial landmarks representing eyes depicted on the portion of the face;

determining a second distance between facial landmarks of the set of facial landmarks representing the eyes and a nose depicted on the portion of the face;

determining a third distance between facial landmarks of the set of facial landmarks representing the eyes and a mouth depicted on the portion of the face; and

determining a fourth distance between facial landmarks of the set of facial landmarks representing the eyes and a chin depicted on the portion of the face.

4. The method of claim 2, wherein determining the one or more characteristics further comprises:

determining a gender of the portion of the face based on the one or more distances between the two or more facial landmarks.

5. The method of claim 2, wherein determining the one or more characteristics further comprises:

determining a race identifier of the portion of the face based on the one or more distances between the two or more facial landmarks; and

determining a skin color by identifying an area of interest on the portion of the face and extracting an average color depicted within the area of interest.

6. The method of claim 2, wherein determining the one or more characteristics further comprises:

determining a jaw shape of the portion of the face based on the set of facial landmarks and the one or more distances between the two or more facial landmarks; and

fitting a polyline to the jaw shape.

7. The method of claim 2, wherein determining the one or more characteristics further comprises:

identifying one or more iris within the portion of the face; and

determining a prevailing color for the one or more iris.

8. The method of claim 7, wherein identifying the one or more iris further comprises:

determining a shape of one or more eyes surrounding the one or more iris;

determining a height of the shape based on the set of facial landmarks; and

determining an iris dimension based on the height of the shape of the one or more eyes.

9. The method of claim 2, wherein determining the one or more characteristics further comprises:

determining one or more eyebrow region of the portion of the face; and

identifying one or more shape of the one or more eyebrow region.

10. The method of claim 2, wherein determining the one or more characteristics further comprises:

determining a lip region of the portion of the face;

identifying a shape of one or more lips within the lip region; and

identifying a prevailing color of the lip region.

11. The method of claim 1, wherein:

the hair volume is determined by;

identifying a first portion of the set of facial landmarks that represents an outline of the face; and

computing a distance the hair region extends from the identified portion of the set of facial landmarks that represents the outline of the face to an outer opposing edge of the hair region; and

the hair length is determined based on the hair volume and by;

identifying a second portion of the set of facial landmarks that represents a chin; and

computing a distance from the hair region to the second portion of the set of facial landmarks that represents the chin.

12. The method of claim 1, wherein determining the one or more characteristics comprises:

selecting a region of the portion of the face where obstructions commonly appear;

determining whether an obstruction is present in the selected region using one or more of the set of facial landmarks; and

in response to determining that the obstruction is present, matching the obstruction to a template selected from a set of templates.

13. A system, comprising:

one or more processors; and

a non-transitory processor-readable storage medium coupled to the one or more processors and storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

receiving, by the one or more processors, one or more two-dimensional images depicting at least a portion of a face of a first user;

detecting, by the one or more processors, the portion of the face depicted within the one or more two-dimensional images;

identifying a set of facial landmarks within the portion of the face depicted within the one or more two-dimensional images;

in response to detecting the portion of the face and the set of facial landmarks, determining one or more characteristics representing the portion of the face depicted in the one or more two-dimensional images, the determining comprising;

identifying a set of colors within a hair region of the face;

determining a hair texture for the hair region based on variations in the set of colors that indicate edges, objects or shapes within the hair associated with the hair texture;

comparing a dimension of the hair region to one or more of the set of facial landmarks; and

determining hair length or hair volume based on the comparison of the dimension of the hair region to the one or more of the set of facial landmarks;

based on the one or more characteristics and the set of facial landmarks, generating a representation of a face for the at least one portion of the face depicted in the one or more images;

generating one or more sticker graphics incorporating the representation of the face, the generating of the one or more sticker graphics comprising generating a scaled version of the representation corresponding to a template graphic; and

generating, by a user device of the first user, a message, directed to a second user, that includes the one or more sticker graphics incorporating the representation of the face of the first user.

14. The system of claim 13, wherein identifying the set of facial landmarks further comprises:

determining a first distance between facial landmarks of the set of facial landmarks representing eyes depicted on the portion of the face;

determining a second distance between facial landmarks of the set of facial landmarks representing the eyes and a nose depicted on the portion of the face;

determining a third distance between facial landmarks of the set of facial landmarks representing the eyes and a mouth depicted on the portion of the face; and

determining a fourth distance between facial landmarks of the set of facial landmarks representing the eyes and a chin depicted on the portion of the face.

15. The system of claim 14, wherein determining the one or more characteristics further comprises:

determining a gender of the portion of the face based on one or more of the first distance, the second distance, the third distance, and the fourth distance.

16. A non-transitory processor-readable storage medium storing processor executable instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

receiving, by one or more processors, one or more two-dimensional images depicting at least a portion of a face of a first user;

detecting, by the one or more processors, the portion of the face depicted within the one or more two-dimensional images;

identifying a set of facial landmarks within the portion of the face depicted within the one or more two-dimensional images;

in response to detecting the portion of the face and the set of facial landmarks, determining one or more characteristics representing the portion of the face depicted in the one or more two-dimensional images, the determining comprising;

identifying a set of colors within a hair region of the face;

determining a hair texture for the hair region based on variations in the set of colors that indicate edges, objects or shapes within the hair associated with the hair texture;

comparing a dimension of the hair region to one or more of the set of facial landmarks; and

determining hair length or hair volume based on the comparison of the dimension of the hair region to the one or more of the set of facial landmarks;

based on the one or more characteristics and the set of facial landmarks, generating a representation of a face for the at least one portion of the face depicted in the one or more images;

generating one or more sticker graphics incorporating the representation of the face, the generating of the one or more sticker graphics comprising generating a scaled version of the representation corresponding to a template graphic; and

generating, by a user device of the first user, a message, directed to a second user, that includes the one or more sticker graphics incorporating the representation of the face of the first user.

17. The non-transitory processor-readable storage medium of claim 16, wherein the operations further comprise:

determining a first distance between facial landmarks of the set of facial landmarks representing eyes depicted on the portion of the face;

determining a second distance between facial landmarks of the set of facial landmarks representing the eyes and a nose depicted on the portion of the face;

determining a third distance between facial landmarks of the set of facial landmarks representing the eyes and a mouth depicted on the portion of the face; and

determining a fourth distance between facial landmarks of the set of facial landmarks representing the eyes and a chin depicted on the portion of the face.

18. The non-transitory processor-readable storage medium of claim 17, wherein determining the one or more characteristics further comprises:

determining a gender of the portion of the face based on one or more of the first distance, the second distance, the third distance, and the fourth distance.

19. The non-transitory processor-readable storage medium of claim 17, wherein determining the one or more characteristics further comprises:

identifying one or more iris within the portion of the face;

determining a shape of one or more eyes surrounding the one or more iris;

determining a height of the shape based on the set of facial landmarks;

determining an iris dimension based on the height of the shape of the one or more eyes; and

determining a prevailing color for the one or more iris.