IMAGE ASSESSMENT USING DEEP CONVOLUTIONAL NEURAL NETWORKS

• US 20160035078A1
• Filed: 07/30/2014
• Published: 02/04/2016
• Est. Priority Date: 07/30/2014
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: LIN, Zhe, JIN, Hailin, YANG, Jianchao
• Granted Patent: US 9,536,293 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries

Abstract

Deep convolutional neural networks receive local and global representations of images as inputs and learn the best representation for a particular feature through multiple convolutional and fully connected layers. A double-column neural network structure receives each of the local and global representations as two heterogeneous parallel inputs to the two columns. After some layers of transformations, the two columns are merged to form the final classifier. Additionally, features may be learned in one of the fully connected layers. The features of the images may be leveraged to boost classification accuracy of other features by learning a regularized double-column neural network.

Claims

1. A non-transitory computer storage medium comprising computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:

implementing a deep convolutional neural network that is trained to learn and classify image features for a set of images;

receiving an image from the set of images;

extracting a local image representation of the image as one or more fine-grained inputs to the deep convolutional neural network;

calculating a probability of each input being assigned to a class for a particular feature;

averaging results associated with each input associated with the image; and

selecting the class with the highest probability.

2. The non-transitory computer storage medium of claim 1, further comprising extracting a global image representation of the image as one or more global inputs to the deep convolutional neural network comprising a first column and a second column, the first column associated with the global input and the second column associated with the fine-grained input.

3. The non-transitory computer storage medium of claim 2, further comprising merging the first column and the second column to form a deep neural network classifier.

4. The non-transitory computer storage medium of claim 2, further comprising resizing the image to create the global image representation.

5. The non-transitory computer storage medium of claim 2, further comprising resizing the image by warping the image into a normalized input with a fixed size.

6. The non-transitory computer storage medium of claim 2, further comprising resizing the image by normalizing its shorter side to a normalized input with a fixed length s and center-cropping the normalized input to generate a s x s x 3 input.

7. The non-transitory computer storage medium of claim 2, further comprising resizing the image by normalizing a longer side of the image to a fixed length s and generating a normalized input of a fixed size s x s x 3 by padding border pixels with zero.

8. The non-transitory computer storage medium of claim 1, further comprising randomly cropping the image into a normalized input with a fixed size to create the local image representation, the local image representation preserving details of the image in original high-resolution format.

9. The non-transitory computer storage medium of claim 2, wherein an architecture associated with each column in the double-column convolutional neural network is the same for each column.

10. The non-transitory computer storage medium of claim 2, wherein an architecture associated with each column in the double-column convolutional neural network is different for each column.

11. The non-transitory computer storage medium of claim 2, further comprising adding one or more additional columns with additional normalized inputs to form a multi-column convolutional neural network.

12. The non-transitory computer storage medium of claim 2, wherein an architecture associated with each column in the deep convolutional neural network comprises at least four convolutional layers and at least two fully-connected layers.

13. The non-transitory computer storage medium of claim 12, further comprising extracting one or more features from the image at one of the fully-connected layers.

14. The non-transitory computer storage medium of claim 12, further comprising replacing a last layer of the deep convolutional neural network with a regression.

15. The non-transitory computer storage medium of claim 1, wherein the features include aesthetics, style, or scene.

16. A computer-implemented method comprising:

implementing a double-column deep convolutional neural network (DCNN) that is trained to learn and classify features for a set of images;

extracting a global image representation of an image as a global input to a first column of the DCNN;

extracting a local image representation of the image as a fine-grained input to a second column of the DCNN, the first column having at least one layer that is independent from at least one layer of the second column;

merging at least one layer of the first column with at least one layer of the second column into a fully connected layer;

jointly training weights associated with the fully connected layer; and

classifying at least one feature for the image.

17. The method of claim 16, further comprising automatically discovering global and local features of an image from the fully connected layer and a layer immediately preceding the fully connected layer.

18. The method of claim 16, further comprising back propagating error in each column with stochastic gradient descent.

19. The method of claim 16, further comprising adding one or more additional columns with additional normalized inputs to form a multi-column convolutional neural network.

20. A computerized system comprising:

one or more processors; and

one or more computer storage media storing computer-useable instructions that, when used by the one or more processors, cause the one or more processors to;

implement a double-column deep convolutional neural network (DCNN) to train the DCNN to learn and classify features for a set of images;

extract a global image representation of an image as a global input to a first column of the DCNN;

extract a local image representation of the image as a fine-grained input to a second column of the DCNN;

merge at least one layer of the first column with at least one layer of the second column into a fully connected layer; and

learn or classify at least one feature for the image.