System for Beauty, Cosmetic, and Fashion Analysis

• US 20170076474A1
• Filed: 02/23/2015
• Published: 03/16/2017
• Est. Priority Date: 02/23/2014
• Status: Active Grant

• Current Assignee: Northeastern University
• Original Assignee: Northeastern University
• Inventors: FU, Yun, WANG, Shuyang
• Granted Patent: US 10,339,685 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

A system and method are provided to detect, analyze and digitally remove makeup from an image of a face. An autoencoder-based framework is provided to extract attractiveness-aware features to perform an assessment of facial beauty.

Claims

1. A system for analyzing an image of a human face for the presence of makeup, comprising:

one or more processors and memory, including a dataset comprising images of human faces, the images comprising facial images of multiple human subjects, and including multiple images associated with a single human subject showing steps of makeup application including a face with no makeup, a face with an intermediate stage of makeup application, and a face with a final makeup application;

the one or more processors trained using the dataset to predict an image of a human face without makeup from an input image of a human face wearing makeup;

machine-readable instructions stored in the memory, that upon execution by the one or more processors cause the system to carry out operations comprising;

receiving from an input device an input image of a human face wearing makeup;

detecting the presence of the makeup on the input image;

decomposing the input image to remove the makeup from the input image by applying a mapping from makeup features to non-makeup features; and

providing to an output device an output image of the human face with the makeup removed from the image.

2. The system of claim 1, further comprising categorizing the makeup on the input image into a category, wherein the categories include one or more of everyday makeup, regular makeup, fashion makeup, fashion photography makeup, fashion runway makeup, television makeup, film makeup, theatrical makeup, stage makeup, special effects makeup, airbrushed makeup, special events makeup, and high definition makeup.

4. The system of claim 1, wherein the detecting step further comprises detecting the presence of the makeup on one or more facial regions, the facial regions comprising an eye region, an eyebrow region, a lip region, and a global skin region.

5. The system of claim 1, wherein:

the detecting step further comprises detecting one or more perceptual effects of the makeup, the perceptual effects comprising skin color, eye shape, lip shape, skin texture, skin smoothness, and skin highlights,

the detected perceptual effects are classified into makeup items by the one or more processors trained with the dataset of human faces; and

the one or more processors are trained with locality-constrained low-rank dictionary learning, a support vector machine classifier, or an adaptive boosting classifier.

8. The system of claim 5, wherein,

the eye shape, the lip shape, and the skin texture are detected by one or more edge detection filters:

the skin texture is characterized by determining local binary patterns for various pixel cells on the image;

the skin smoothness is characterized by image intensity values at various pixels on the image; and

the skin highlights are characterized by determining dichromatic reflections of the skin.

12. The system of claim 1, wherein the dataset includes images separated by facial regions, the facial regions comprising one or more of an eye region, an eyebrow region, a lip region, and a global skin region.

13. The system of claim 1, further comprising one or more of,

the one or more processors are trained by sequential dictionary learning using a set of sub-dictionaries learned from the dataset, and the step of decomposing the input image to remove the makeup comprises applying a projection matrix through at least a portion of the set of sub-dictionaries,

the step of decomposing the input image comprises finding a nearest neighbor image in the dataset and removing makeup under the guidance of the dataset,

the step of decomposing the input image comprises mapping makeup features on the input image to non-makeup features on the output image,

the one or more processors are trained with locality-constrained low-rank dictionary learning, a semi-coupled dictionary learning method, a Bayesian inference method, a subspace learning method, a sparse representation method, or a deep learning method.

17. The system of claim 1, further comprising, prior to the step of detecting the presence of makeup on the input image, the steps of locating fiducial landmarks on the facial image, warping the facial image into a canonical form, and splitting the facial image into facial regions,

after the step of decomposing the input image, warping the facial image back and blending to replace a reconstructed part of the image on an original image, and adding a textural detail comprising original wrinkles to the image.

22. The system of claim 1, wherein,

the input device comprises a scanner, a camera, a computer, a mobile device, or a further processor,

the output device comprises a video display device, a computer monitor, a mobile device, a printer, a facial recognition system, or a security system;

and

the one or more processor and the memory are disposed on a computer, a server, or a mobile device.

3. (canceled)

6-7. -7. (canceled)

9-11. -11. (canceled)

14-16. -16. (canceled)

18-21. -21. (canceled)

23-24. -24. (canceled)

25. A method for analyzing an image of a human face for the presence of makeup, comprising:

receiving an input image of a human face wearing makeup at a computer comprising one or more processors and memory, including a dataset comprising images of human faces, the images comprising facial images of multiple human subjects, and including multiple images associated with a single human subject showing steps of makeup application including a face with no makeup, a face with an intermediate stage of makeup application, and a face with a final makeup application, the one or more processors trained using the dataset to predict an image of a human face without makeup from an input image of a human face wearing makeup;

detecting the presence of the makeup on the input image by reference to the dataset;

decomposing the input image to remove the makeup from the input image by applying a mapping from makeup features to non-makeup features in the dataset; and

providing to an output device an output image of the human face with the makeup removed from the image.

26. The method of claim 25, further comprising categorizing the makeup on the input image into a category, wherein the categories include one or more of everyday makeup, regular makeup, fashion makeup, fashion photography makeup, fashion runway makeup, television makeup, film makeup, theatrical makeup, stage makeup, special effects makeup, airbrushed makeup, special events makeup, and high definition makeup.

28. The method of claim 25, wherein the detecting step further comprises detecting the presence of the makeup on one or more facial regions, the facial regions comprising an eye region, an eyebrow region, a lip region, and a global skin region.

29. The method of claim 25, wherein,

the detecting step further comprises detecting one or more perceptual effects of the makeup, the perceptual effects comprising skin color, eye shape, lip shape, skin texture, skin smoothness, and skin highlights;

the detected perceptual effects are classified into makeup items by the one or more processors trained with the dataset of human faces; and

the one or more processors are trained with locality-constrained low-rank dictionary learning, a support vector machine classifier, or an adaptive boosting classifier.

32. The method of claim 29, wherein,

the eye shape, the lip shape, and the skin texture are detected by one or more edge detection filters,

the skin texture is characterized by determining local binary patterns for various pixel cells on the image;

the skin smoothness is characterized by image intensity values at various pixels on the image; and

the skin highlights are characterized by determining dichromatic reflections of the skin.

36. The method of claim 25, wherein the dataset includes images separated by facial regions, the facial regions comprising one or more of an eye region, an eyebrow region, a lip region, and a global skin region.

37. The method of claim 25, wherein the one or more processors are trained by sequential dictionary learning using a set of sub-dictionaries learned from the dataset, and the step of decomposing the input image to remove the makeup comprises applying a projection matrix through at least a portion of the set of sub-dictionaries.

38. The method of claim 25, wherein the step of decomposing the input image comprises finding a nearest neighbor image in the dataset and removing makeup under the guidance of the dataset.

39. The method of claim 25, wherein the step of decomposing the input image comprises mapping makeup features on the input image to non-makeup features on the output image.

40. The method of claim 25, further comprising training the one or more processors with locality-constrained low-rank dictionary learning, a semi-coupled dictionary learning method, a Bayesian inference method, a subspace learning method, a sparse representation method, or a deep learning method.

41. The method of claim 25, further comprising, prior to the step of detecting the presence of makeup on the input image, locating fiducial landmarks on the facial image, warping the facial image into a canonical form, and splitting the facial image into facial regions.

42. The method of claim 41, further comprising, after the step of decomposing the input image, warping the facial image back and blending to replace a reconstructed part of the image on an original image.

43. The method of claim 25, further comprising, after the decomposing step, adding a textural detail comprising original wrinkles to the image.

44. The method of claim 25, further comprising evaluating the input makeup image by determining suitability of a makeup style for one or more personal characteristics, for an event, or for an occasion, and wherein the memory further includes a dataset of information about cosmetic products and instructions to provide a selection of cosmetic products to emulate a makeup face.

27. (canceled)

30-31. -31. (canceled)

33-35. -35. (canceled)

45-53. -53. (canceled)

54. A method for providing an assessment of facial attractiveness, comprising:

providing one or more processors and memory, including a first auto-encoder trained with one or more visual descriptors of more attractive faces and a second auto-encoder trained with one or more visual descriptors of less attractive faces;

receiving from an input device an input image of a human face;

extracting low-level features from the input image of the human face,

inputting the low-level features to the first autoencoder and to the second autoencoder;

determining a first output from the first autoencoder and a second output from the second autoencoder;

comparing a difference between the first output and the second output, the difference comprising a value representative of attractiveness of the human face; and

outputting the value representative of attractiveness to an output device.

55-58. -58. (canceled)