Optimal image transformation based on professionalism score of subject

• US 10,043,254 B2
• Filed: 04/14/2016
• Issued: 08/07/2018
• Est. Priority Date: 04/14/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Merhav, Uri, Shacham, Dan
• Primary Examiner(s): Park, Edward
• Application Number:
  US15/098,950
• Publication Number:
  US 20170301077A1
• Time in Patent Office: 845 Days
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 30/19173 Classification techniques

Abstract

In an example embodiment, an image transformation is automatically performed on a digital image to improve perceived professionalism of a subject of the image. A machine learning algorithm is utilized to generate a professionalism score for the digital image, the utilizing a machine learning algorithm comprising: a training mode where a plurality of sample images with labeled professionalism scores are used to train a classification function in a model that produces as professionalism score as output; an analysis mode where the model is used to generate a professionalism score for the digital image. Then the professionalism score is used as an input to a continuous variable optimization algorithm to determine an optimum version of the digital image from a plurality of possible versions of the digital image on which one or more image transformations have been performed, using the classification function.

Claims

1. A computerized method for automatically performing an image transformation on a digital image to improve perceived professionalism of a subject of the image, the method comprising:

utilizing a machine learning algorithm to generate a professionalism score for the digital image, wherein the professionalism score indicates a perceived professionalism of a human depicted in the digital image, the utilizing a machine learning algorithm comprising;

a training mode where a plurality of sample images with labeled professionalism scores are used to train a classification function in a model that produces as professionalism score as output;

an analysis mode where the model is used to generate a professionalism score for the digital image; and

using the professionalism score for the digital image as an input to a continuous variable optimization algorithm to determine an optimum version of the digital image from a plurality of possible versions of the digital image on which one or more image transformations have been performed, using the classification function, wherein the continuation variable optimization algorithm uses a deep convolutional neural network (DCNN) by passing the digital image to a convolutional layer, generating output, passing the output from the convolutional layer to a nonlinearity layer, generating output, passing the output from the nonlinearity layer to a pooling layer, generating output, and passing output from the nonlinearity layer to a classification layer.

2. The method of claim 1, wherein the classification function contains one or more weights, and wherein the training of the classification function incudes modifying the one or more weights.

3. The method of claim 1, wherein the one or more image transformations includes image rotation.

4. The method of claim 1, wherein the one or more image transformations includes lighting compensation.

5. The method of claim 1, wherein the one or more image transformations includes edge enhancing filtering.

6. The method of claim 1, wherein the one or more image transformations includes edge enhancing filtering.

7. The method of claim 1, wherein the one or more image transformations includes spatially varying blur filtering.

8. The method of claim 1, wherein the one or more image transformations includes at least two image transformations selected from the group consisting of:

image rotation;

lighting compensation;

edge enhancing filtering;

edge enhancing filtering; and

spatially varying blur filtering.

9. The method of claim 1, wherein the continuous variable optimization algorithm is a derivative-free optimization algorithm.

10. The method of claim 1, wherein the continuous variable optimization algorithm is a constrained optimization algorithm.

11. A non-transitory machine-readable storage medium comprising instructions, which when implemented by one or more machines, cause the one or more machines to perform operations for automatically performing an image transformation on a digital image to improve perceived professionalism of a subject of the image, the operations comprising:

utilizing a machine learning algorithm to generate a professionalism score for the digital image, wherein the professionalism score indicates a perceived professionalism of a human depicted in the digital image, the utilizing a machine learning algorithm comprising;

a training mode where a plurality of sample images with labeled professionalism scores are used to train a classification function in a model that produces as professionalism score as output;

an analysis mode where the model is used to generate a professionalism score for the digital image; and

using the professionalism score for the digital image as an input to a continuous variable optimization algorithm to determine an optimum version of the digital image from a plurality of possible versions of the digital image on which one or more image transformations have been performed, using the classification function, wherein the continuation variable optimization algorithm uses a deep convolutional neural network (DCNN) by passing the digital image to a convolutional layer, generating output, passing the output from the convolutional layer to a nonlinearity layer, generating output, passing the output from the nonlinearity layer to a pooling layer, generating output, and passing output from the nonlinearity layer to a classification layer.

12. The non-transitory machine-readable storage medium of claim 11, wherein the classification function contains one or more weights, and wherein the training of the classification function incudes modifying the one or more weights.

13. The non-transitory machine-readable storage medium of claim 11, wherein the one or more image transformations includes image rotation.

14. The non-transitory machine-readable storage medium of claim 11, wherein the one or more image transformations includes lighting compensation.

15. The non-transitory machine-readable storage medium of claim 11, wherein the one or more image transformations includes edge enhancing filtering.

16. The non-transitory machine-readable storage medium of claim 11, wherein the one or more image transformations includes edge enhancing filtering.

17. The non-transitory machine-readable storage medium of claim 11, wherein the one or more image transformations includes spatially varying blur filtering.

18. The non-transitory machine-readable storage medium of claim 11, wherein the one or more image transformations includes at least two image transformations selected from the group consisting of:

image rotation;

lighting compensation;

edge enhancing filtering;

edge enhancing filtering; and

spatially varying blur filtering.

19. The non-transitory machine-readable storage medium of claim 11, wherein the continuous variable optimization algorithm is a derivative-free optimization algorithm.

20. The non-transitory machine-readable storage medium of claim 11, wherein the continuous variable optimization algorithm is a constrained optimization algorithm comprising:

training a DCNN by;

inputting a plurality of sample images to the DCNN, each of the sample images having been labeled with a professionalism score, the inputting including, for each sample image;

passing the image to a convolutional layer of the DCNN, the convolutional layer comprising one or more filters having dynamically adjustable weights, the one or more filters configured to filter the image to produce an output volume for the corresponding image, the output volume comprising a different feature map for each of the one or more filters;

passing the output volume from the convolutional layer through a nonlinearity layer, the nonlinearity layer applying a nonlinearity function to the output volume from the convolutional layer;

passing the output volume from the nonlinearity layer through a pooling layer, the pooling layer lowering spatial dimensions of the output volume from the nonlinearity layer;

passing the output volume from the pooling layer through a classification layer, the classification layer comprising a specialized convolutional layer having a filter designed to output a professionalism score for the image based on the output volume from the pooling layer; and

passing the image through a loss layer, the loss layer applying a loss function to the image, resulting an in indication of a level of error in the professionalism score for the image from the classification layer in comparison to the professionalism score from the label of the image;

determining whether a combination of the levels of error for the plurality of sample images transgresses a preset threshold; and

in response to a determination that the combination of the levels of error transgresses a preset threshold, updating weights of the one or more filters in the convolutional layers of the DCNN to reduce the combination of the levels of error and repeating the training of the DCNN using a different plurality of sample images and the updated weights.