Deep convolutional neural network prediction of image professionalism

• US 9,904,871 B2
• Filed: 04/14/2016
• Issued: 02/27/2018
• Est. Priority Date: 04/14/2016
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Merhav, Uri, Shacham, Dan
• Primary Examiner(s): MOTSINGER, SEAN T
• Application Number:
  US15/098,906
• Publication Number:
  US 20170300785A1
• Time in Patent Office: 684 Days
• CPC Class Codes: G06F 17/11 for solving equations , e.g. nonlinear equations, general mathematical optimization problems optimization specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06F 18/2433 Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 3/042 Knowledge-based neural networks; Logical representations of neural networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/7747 Organisation of the process, e.g. bagging or boosting

Abstract

In an example embodiment, a deep convolutional neural network (DCNN) is created to assign a professionalism score to an input image. The professionalism score indicates a perceived professionalism of a subject of the input image. The DCNN is designed to automatically learn features of images relevant to the professionalism through a training process.

Claims

1. A computerized method of training and utilizing a deep convolutional neural network (DCNN) to gauge professionalism of a subject in a digital image, the method comprising:

training the DCNN by;

inputting a plurality of sample images to the DCNN, each of the sample images having been labeled with a professionalism score, the inputting including, for each sample image;

passing the image to a convolutional layer of the DCNN, the convolutional layer comprising one or more filters having dynamically adjustable weights, the one or more filters configured to filter the image to produce an output volume for the corresponding image, the output volume comprising a different feature map for each of the one or more filters;

passing the output volume from the convolutional layer through a nonlinearity layer, the nonlinearity layer applying a nonlinearity function to the output volume from the convolutional layer;

passing the output volume from the nonlinearity layer through a pooling layer, the pooling layer lowering spatial dimensions of the output volume from the nonlinearity layer;

passing the output volume from the pooling layer through a classification layer, the classification layer comprising a specialized convolutional layer having a filter designed to output a professionalism score for the image based on the output volume from the pooling layer; and

passing the image through a loss layer, the loss layer applying a loss function to the image, resulting an in indication of a level of error in the professionalism score for the image from the classification layer in comparison to the professionalism score from the label of the image;

determining whether a combination of the levels of error for the plurality of sample images transgresses a preset threshold; and

in response to a determination that the combination of the levels of error transgresses a preset threshold, updating weights of the one or more filters in the convolutional layers of the DCNN to reduce the combination of the levels of error and repeating the training of the DCNN using a different plurality of sample images and the updated weights;

generating a professionalism score for the digital image by;

inputting the digital image to the DCNN, including;

passing the image to the convolutional layer, generating output;

passing the output from the convolutional layer to the nonlinearity layer, generating output;

passing the output from the nonlinearity layer to the pooling layer, generating output; and

passing output from the nonlinearity layer to the classification layer, generating a professionalism score for the digital image.

2. The method of claim 1, further comprising:

preprocessing the plurality of sample images and the digital image to normalize color space and size of each image.

3. The method of claim 1, further comprising:

using the professionalism score to perform image transformation on the digital image.

4. The method of claim 1, further comprising:

using the professionalism score to perform cropping on the digital image.

5. The method of claim 1, wherein the DCNN comprises multiple stages, each stage containing a different convolutional layer, nonlinearity layer, and pooling layer.

6. The method of claim 1, wherein the loss function is static.

7. The method of claim 1, wherein the loss function is a sum squared error function.

8. A system comprising:

a processor;

a computer readable medium having instructions stored there on, which, when executed by the processor, cause the system to;

train a DCNN by;

inputting a plurality of sample images to the DCNN, each of the sample images having been labeled with a professionalism score, the inputting including, for each sample image;

passing the image to a convolutional layer of the DCNN, the convolutional layer comprising one or more filters having dynamically adjustable weights, the one or more filters configured to filter the image to produce an output volume for the corresponding image, the output volume comprising a different feature map for each of the one or more filters;

passing the output volume from the convolutional layer through a nonlinearity layer, the nonlinearity layer applying a nonlinearity function to the output volume from the convolutional layer;

passing the output volume from the nonlinearity layer through a pooling layer, the pooling layer lowering spatial dimensions of the output volume from the nonlinearity layer;

passing the output volume from the pooling layer through a classification layer, the classification layer comprising a specialized convolutional layer having a filter designed to output a professionalism score for the image based on the output volume from the pooling layer; and

passing the image through a loss layer, the loss layer applying a loss function to the image, resulting an in indication of a level of error in the professionalism score for the image from the classification layer in comparison to the professionalism score from the label of the image;

determining whether a combination of the levels of error for the plurality of sample images transgresses a preset threshold; and

in response to a determination that the combination of the levels of error transgresses a preset threshold, updating weights of the one or more filters in the convolutional layers of the DCNN to reduce the combination of the levels of error and repeating the training of the DCNN using a different plurality of sample images and the updated weights;

generate a professionalism score for the digital image by;

inputting the digital image to the DCNN, including;

passing the image to the convolutional layer, generating output;

passing the output from the convolutional layer to the nonlinearity layer, generating output;

passing the output from the nonlinearity layer to the pooling layer, generating output; and

passing output from the nonlinearity layer to the classification layer, generating a professionalism score for the digital image.

9. The system of claim 8, wherein the instructions further cause the system to:

preprocess the plurality of sample images and the digital image to normalize color space and size of each image.

10. The system of claim 8, wherein the instructions further cause the system to:

use the professionalism score to perform image transformation on the digital image.

11. The system of claim 8, further comprising:

using the professionalism score to perform cropping on the digital image.

12. The system of claim 8, wherein the DCNN comprises multiple stages, each stage containing a different convolutional layer, nonlinearity layer, and pooling layer.

13. The system of claim 8, wherein the loss function is static.

14. The system of claim 8, wherein the loss function is a sum squared error function.

15. A non-transitory machine-readable storage medium comprising instructions, which when implemented by one or more machines, cause the one or more machines to perform operations comprising:

training a DCNN by;

inputting a plurality of sample images to the DCNN, each of the sample images having been labeled with a professionalism score, the inputting including, for each sample image;

passing the image to a convolutional layer of the DCNN, the convolutional layer comprising one or more filters having dynamically adjustable weights, the one or more filters configured to filter the image to produce an output volume for the corresponding image, the output volume comprising a different feature map for each of the one or more filters;

passing the output volume from the convolutional layer through a nonlinearity layer, the nonlinearity layer applying a nonlinearity function to the output volume from the convolutional layer;

passing the output volume from the nonlinearity layer through a pooling layer, the pooling layer lowering spatial dimensions of the output volume from the nonlinearity layer;

passing the output volume from the pooling layer through a classification layer, the classification layer comprising a specialized convolutional layer having a filter designed to output a professionalism score for the image based on the output volume from the pooling layer; and

passing the image through a loss layer, the loss layer applying a loss function to the image, resulting an in indication of a level of error in the professionalism score for the image from the classification layer in comparison to the professionalism score from the label of the image;

determining whether a combination of the levels of error for the plurality of sample images transgresses a preset threshold; and

in response to a determination that the combination of the levels of error transgresses a preset threshold, updating weights of the one or more filters in the convolutional layers of the DCNN to reduce the combination of the levels of error and repeating the training of the DCNN using a different plurality of sample images and the updated weights;

generating a professionalism score for the digital image by;

inputting the digital image to the DCNN, including;

 

passing the image to the convolutional layer, generating output;

 

passing the output from the convolutional layer to the nonlinearity layer, generating output;

 

passing the output from the nonlinearity layer to the pooling layer, generating output; and

 

passing output from the nonlinearity layer to the classification layer, generating a professionalism score for the digital image.

16. The non-transitory machine-readable of claim 15, further comprising:

preprocessing the plurality of sample images and the digital image to normalize color space and size of each image.

17. The non-transitory machine-readable of claim 15, further comprising:

using the professionalism score to perform image transformation on the digital image.

18. The non-transitory machine-readable of claim 15, further comprising:

using the professionalism score to perform cropping on the digital image.

19. The non-transitory machine-readable of claim 15, wherein the DCNN comprises multiple stages, each stage containing a different convolutional layer, nonlinearity layer, and pooling layer.

20. The non-transitory machine-readable of claim 15, wherein the loss function is static.