Generating cartoon images from photos

• US 10,529,115 B2
• Filed: 03/14/2018
• Issued: 01/07/2020
• Est. Priority Date: 03/20/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Sarna, Aaron, Krishnan, Dilip, Cole, Forrester, Mosseri, Inbar
• Primary Examiner(s): Hajnik, Daniel F
• Application Number:
  US15/921,207
• Publication Number:
  US 20180268595A1
• Time in Patent Office: 664 Days
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites

Abstract

A system and method for generating cartoon images from photos are described. The method includes receiving an image of a user, determining a template for a cartoon avatar, determining an attribute needed for the template, processing the image with a classifier trained for classifying the attribute included in the image, determining a label generated by the classifier for the attribute, determining a cartoon asset for the attribute based on the label, and rendering the cartoon avatar personifying the user using the cartoon asset.

Claims

1. A computer-implemented method comprising:

receiving, using one or more computing devices, an image of a user;

determining, using the one or more computing devices, a template for a cartoon avatar;

determining, using the one or more computing devices, attributes needed for the template, wherein the attributes include at least one facial attribute;

processing, using the one or more computing devices, the image with a classifier trained for classifying attributes included in the image, wherein the classifier includes a shallow neural network, wherein the shallow neural network is an attribute specific classifier and is coupled to a selected layer of a pre-trained deep neural network model, and wherein the selected layer of the pre-trained deep neural network model is selected by determining that the selected layer provides discrimination for a selected attribute from different training set images and is invariant to parameters that do not affect the selected attribute;

determining, using the one or more computing devices, labels generated by the classifier for the attributes included in the image;

determining, using the one or more computing devices, cartoon assets for the attributes included in the image, based on the labels; and

rendering, using the one or more computing devices, the cartoon avatar personifying the user using the cartoon assets by;

organizing one or more of the cartoon assets into a position relative to other cartoon assets corresponding to the at least one facial attribute in the template;

stitching and blending edges of the cartoon assets; and

rendering the cartoon avatar based on the organized and stitched cartoon assets.

2. The computer-implemented method of claim 1, wherein determining the cartoon assets for the attributes included in the image based on the labels comprises:

determining a set of cartoon assets associated with the attributes included in the image, wherein each cartoon asset has an associated label;

mapping the labels to the set of cartoon assets;

determining whether the cartoon assets are available within the set of cartoon assets based on the mapping; and

responsive to determining that the cartoon assets are available within the set of cartoon assets, selecting, from the set of cartoon assets, the cartoon assets for the rendering.

3. The computer-implemented method of claim 2, wherein determining the set of cartoon assets associated with the attributes included in the image comprises:

determining a cartoon style of the cartoon avatar; and

determining the set of cartoon assets associated with the attributes included in the image based on the cartoon style.

4. The computer-implemented method of claim 2, wherein determining the set of cartoon assets associated with the attributes included in the image comprises:

determining a cartoon expression of the cartoon avatar; and

determining the set of cartoon assets associated with the attributes included in the image based on the cartoon expression.

5. The computer-implemented method of claim 2, further comprising:

responsive to determining that one of the cartoon assets is unavailable within the set of cartoon assets, selecting a default cartoon asset for the rendering.

6. The computer-implemented method of claim 1, wherein the parameters that do not affect the selected attribute include at least one of an illumination in the image, a pose of the user in the image, or a background of the image.

7. The computer-implemented method of claim 1, further comprising:

presenting the cartoon avatar on a display of a user device;

receiving user input on the display;

editing the cartoon assets for attributes included in the cartoon avatar based on the user input; and

re-rendering the cartoon avatar based on the edited cartoon assets.

8. A computer program product comprising a non-transitory computer readable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to perform operations comprising:

receiving an image of a user;

determining a template for a cartoon avatar;

determining attributes needed for the template;

processing the image with a classifier trained for classifying the attributes included in the image, wherein the attributes include at least one facial attribute, wherein the classifier includes a shallow neural network, wherein the shallow neural network is an attribute specific classifier and is coupled to a selected layer of a pre-trained deep neural network model, and wherein the selected layer of the pre-trained deep neural network model is selected by determining that the selected layer provides discrimination for a selected attribute from different training set images and is invariant to parameters that do not affect the selected attribute;

determining labels generated by the classifier for the attributes included in the image;

determining cartoon assets for the attributes included in the image based on the labels; and

rendering the cartoon avatar personifying the user using the cartoon assets by;

organizing one or more of the cartoon assets into a position relative to other cartoon assets corresponding to the at least one facial attribute in the template;

stitching and blending edges of the cartoon assets; and

rendering the cartoon avatar based on the organized and stitched cartoon assets.

9. The computer program product of claim 8, wherein determining the cartoon assets for the attributes based on the labels comprises:

determining a set of cartoon assets associated with the attributes included in the image, wherein each cartoon asset has an associated label;

mapping the labels to the set of cartoon assets;

determining whether the cartoon assets are available within the set of cartoon assets based on the mapping; and

responsive to determining that the cartoon assets are available within the set of cartoon assets, selecting, from the set of cartoon assets, the cartoon assets for the rendering.

10. The computer program product of claim 9, wherein determining the set of cartoon assets associated with the attributes comprises:

determining a cartoon style of the cartoon avatar; and

determining the set of cartoon assets associated with the attributes included in the image based on the cartoon style.

11. The computer program product of claim 9, wherein determining the set of cartoon assets associated with the attributes included in the image comprises:

determining a cartoon expression of the cartoon avatar; and

determining the set of cartoon assets associated with the attributes included in the image based on the cartoon expression.

12. The computer program product of claim 9, further comprising:

responsive to determining that one of the cartoon assets is unavailable within the set of cartoon assets, selecting a default cartoon asset for the rendering.

13. The computer program product of claim 8, wherein the at least one facial attribute includes at least one of skin tone, hair color, or facial hair.

14. The computer program product of claim 8, further comprising:

presenting the cartoon avatar on a display of a user device;

receiving user input on the display;

editing the cartoon assets for attributes included in the cartoon avatar based on the user input; and

re-rendering the cartoon avatar based on the edited cartoon assets.

15. A system comprising:

a processor; and

a memory storing instructions that, when executed, cause the system to perform operations comprising;

receiving an image of a user;

determining a template for a cartoon avatar;

determining attributes needed for the template, wherein the attributes include at least one facial attribute;

processing the image with a classifier trained for classifying the attributes included in the image, wherein the classifier includes a shallow neural network, wherein the shallow neural network is an attribute specific classifier and is coupled to a selected layer of a pre-trained deep neural network model, and wherein the selected layer of the pre-trained deep neural network model is selected by determining that the selected layer provides discrimination for a selected attribute from different training set images and is invariant to parameters that do not affect the selected attribute;

determining labels generated by the classifier for the attributes included in the image;

determining cartoon assets for the attributes included in the image based on the labels; and

rendering the cartoon avatar personifying the user using the cartoon assets by;

organizing one or more of the cartoon assets into a position relative to other cartoon assets corresponding to the at least one facial attribute in the template;

stitching and blending edges of the cartoon assets; and

rendering the cartoon avatar based on the organized and stitched cartoon assets.

16. The system of claim 15, wherein to determine the cartoon assets for the attributes based on the labels, the operations further comprise:

determining a set of cartoon assets associated with the attributes included in the image, wherein each cartoon asset has an associated label;

mapping the labels to the set of cartoon assets;

determining whether the cartoon assets are available within the set of cartoon assets based on the mapping; and

responsive to determining that the cartoon assets are available within the set of cartoon assets, selecting, from the set of cartoon assets, the cartoon assets for the rendering.

17. The system of claim 16, wherein to determine the set of cartoon assets associated with the attributes, the operations further comprise:

determining a cartoon style of the cartoon avatar; and

determining the set of cartoon assets associated with the attributes included in the image based on the cartoon style.

18. The system of claim 16, wherein to determine the set of cartoon assets associated with the attributes, the operations further comprise:

determining a cartoon expression of the cartoon avatar; and

determining the set of cartoon assets associated with the attributes included in the image based on the cartoon expression.

19. The system of claim 16, further comprising:

responsive to determining that one of the cartoon assets is unavailable within the set of cartoon assets, selecting a default cartoon asset for the rendering.

20. The system of claim 15, wherein the parameters that do not affect the selected attribute include at least one of an illumination in the image, a pose of the user in the image, or a background of the image.