System for beauty, cosmetic, and fashion analysis

• US 10,726,601 B2
• Filed: 05/17/2019
• Issued: 07/28/2020
• Est. Priority Date: 02/23/2014
• Status: Active Grant

• Current Assignee: Northeastern University
• Original Assignee: Northeastern University
• Inventors: Fu, Yun, Wang, Shuyang
• Primary Examiner(s): Saini, Amandeep
• Application Number:
  US16/415,128
• Publication Number:
  US 20190272659A1
• Time in Patent Office: 438 Days
• Field of Search: 382100
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

A system and method are provided to detect, analyze and digitally remove makeup from an image of a face. An autoencoder-based framework is provided to extract attractiveness-aware features to perform an assessment of facial beauty.

Claims

1. A system for providing an assessment of facial attractiveness, comprising:

one or more processors and memory, including a first auto-encoder trained with one or more visual descriptors of a first set of faces and a second auto-encoder trained with one or more visual descriptors of a second set of faces, the first set of faces being associated with a level of attractiveness that is higher than a level of attractiveness associated with the second set of faces;

machine-readable instruction stored in the memory, that upon execution by the one or more processors cause the system to carry out operations comprising;

receiving from an input device an input image of a human face;

extracting features from the input image of the human face;

inputting the features to the first autoencoder and to the second autoencoder;

determining a first output from the first autoencoder and a second output from the second autoencoder, wherein the second output from the second autoencoder reproduces the input to the second encoder, and the first output from the first autoencoder diverges from the input to the first autoencoder toward a template of a human face associated with a level of attractiveness that is higher than a level of attractiveness associated with the input image;

determining a difference between the first output and the second output, the difference comprising a value representative of attractiveness of the human face; and

outputting the value representative of attractiveness to an output device.

2. The system of claim 1, wherein the features are extracted using visual descriptors selected from one or more of raw pixels, an eigenface, a local binary pattern, a scale invariant feature transform, and a Gabor filter.

3. The system of claim 1, wherein the features are extracted using a first type of visual descriptor or a first patch of the human face to obtain the value representative of attractiveness;

and

the operations further comprise;

extracting further features using a second type of visual descriptor different from the first type of visual descriptor or a second patch of the human face different from the first patch to obtain a second value representative of attractiveness, and

integrating the value and the second value into a fused value representative of attractiveness.

4. The system of claim 1, wherein the extracting the features from the input image of the human face comprises extracting the features from patches of the human face.

5. A method for providing an assessment of facial attractiveness, comprising:

providing a first auto-encoder trained with one or more visual descriptors of a first set of faces and a second auto-encoder trained with one or more visual descriptors of a second set of faces, the first set of faces being associated with a level of attractiveness that is higher than a level of attractiveness associated with the second set of faces, wherein the first auto-encoder and the second auto-encoder are executed on one or more computer processors;

receiving from an input device an input image of a human face;

extracting features from the input image of the human face;

inputting the features to the first autoencoder and to the second autoencoder;

determining a first output from the first autoencoder and a second output from the second autoencoder, wherein the second output from the second autoencoder reproduces the input to the second encoder, and the first output from the first autoencoder diverges from the input to the first autoencoder toward a template of a human face associated with a level of attractiveness that is higher than a level of attractiveness associated with the input image;

determining a difference between the first output and the second output, the difference comprising a value representative of attractiveness of the human face; and

outputting the value representative of attractiveness to an output device.

6. The method of claim 5, wherein the features are extracted using visual descriptors selected from one or more of raw pixels, an eigenface, a local binary pattern, a scale invariant feature transform, and a Gabor filter.

7. The method of claim 5, wherein the features are extracted using a first type of visual descriptor or a first patch of the human face to obtain the value representative of attractiveness;

and

further comprising;

extracting further features using a second type of visual descriptor different from the first type of visual descriptor or a second patch of the human face different from the first patch to obtain a second value representative of attractiveness, and

integrating the value and the second value into a fused value representative of attractiveness.

8. The method of claim 5, wherein the extracting the features from the input image of the human face comprises extracting the features from patches of the human face.