System and method for guiding a user to take a selfie

• US 10,574,883 B2
• Filed: 05/31/2018
• Issued: 02/25/2020
• Est. Priority Date: 05/31/2017
• Status: Active Grant

• Current Assignee: The Procter & Gamble Company, Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: The Procter & Gamble Company
• Inventors: Purwar, Ankur N M N, Matts, Paul Jonathan, Shreve, Matthew Adam, Wu, Wencheng N M N
• Primary Examiner(s): Luo, Kate H
• Application Number:
  US15/993,973
• Publication Number:
  US 20180352150A1
• Time in Patent Office: 635 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30242 Counting objects in image
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 5/445 for displaying additional information H04N5/50 takes precedence

Abstract

Systems and methods for improving the image quality of a selfie. The system and method include analyzing a selfie with image processing modules and selfie quality index modules. The image processing modules process the image using face and landmark detection techniques and face normalization techniques. The selfie quality index modules analyze the selfie for important image quality metrics and provide feature vectors that are used to generate a selfie quality index score. The selfie quality index score can be used to determine whether the selfie meet minimum threshold quality requirement for further analysis by the system for a skin condition.

Claims

1. A system for improving the quality of a selfie, comprising:

a. an image capture device;

b. a computer coupled to the image capture device, the computer comprising a non-transitory memory component with logic stored thereon, wherein the logic comprises

i) an image processing module that causes the computer to locate a face in a selfie captured by the image capture device and provided to the computer, locate at least one landmark facial feature present on the face, and normalize the selfie, and

ii) a selfie quality index (SQI) module that causes the computer to analyze the normalized selfie for a quality metric selected from occlusion, blur, distance-to-camera, facial expression, illumination, and combinations of these, wherein the system uses the analyzed image quality metric to generate an SQI score,

iii) wherein the SQI module comprises a Blur Detection Module in the form of a CNN trained by a blur feature vector formed from at least one of a blur coefficient and a feature extracted from the last layer of an Alexnet CNN, wherein the blur coefficient is generated by taking the differences between Laplacian measures on the normalized image after several consecutive rounds of blurring; and

c. a display coupled to at least one of the image capture device and the computing device.

2. The system of claim 1, wherein the image processing module comprises a Face Detection and Counting Module that isolates at least a portion of the face from background features present in the selfie using a bounding box.

3. The method of claim 2, wherein the SQI module comprises a Distance to Camera Module that determines the distance-to-camera image quality metric by calculating a ratio of pixels in the bounding box to image size.

4. The method of claim 3, wherein the logic causes the computer to communicate to a user that the image capture device is too close when the ratio of pixels in the bounding box to image size is greater than 0.8.

5. The method of claim 3, wherein the logic causes the computer to communicate to a user that the image capture device is too far away when the ratio of pixels in the bounding box to image size is less than 0.4.

6. The system of claim 1, wherein the image processing module comprises a Landmark Detection Module in the form of a convolutional neural network (CNN) to determine if the at least one landmark is present.

7. The method of claim 6, wherein the SQI module comprises a Distance to Camera Module that determines the distance-to-camera image quality metric by comparing a pixel based distance between at least two landmark points in the selfie to a known average distance between said landmark points for a population.

8. The system of claim 1, wherein the image processing module comprises a Face Registration Module that causes the system to normalize the face based on positions of at least 4 facial landmark points.

9. The system of claim 1, wherein the SQI module comprises an Occlusion Module that analyzes the normalized image for the occlusion image quality metric, and wherein the Occlusion Module generates an occlusion feature vector formed by fusing at least one local binary pattern feature with at least one CNN-based Alexnet feature.

10. The method of claim 1, wherein the SQI module comprises an Expression Module that analyzes the normalized selfie for the expression image quality metric, and wherein the Expression Module applies a local binary pattern, histogram of gradients, deep learning technique, or combination of these to determine an expression feature vector.

11. The method of claim 1, wherein the SQI module comprises an Illumination Module that analyzes the normalized selfie for the illumination image quality metric.

12. The method of claim 11, wherein Illumination Module includes at least one sub-module for determining a direction of illumination, an intensity of illumination, contrast, and location of illumination.