Similarity-based detection of prominent objects using deep CNN pooling layers as features

• US 9,767,381 B2
• Filed: 09/22/2015
• Issued: 09/19/2017
• Est. Priority Date: 09/22/2015
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Rodrguez-Serrano, Jos A., Gordo Soldevila, Albert
• Primary Examiner(s): AHMED, SAMIR ANWAR
• Application Number:
  US14/861,386
• Publication Number:
  US 20170083792A1
• Time in Patent Office: 728 Days
• CPC Class Codes: G06F 16/54 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2210/12 Bounding box
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/625 License plates
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A system and method provide object localization in a query image based on a global representation of the image generated with a model derived from a convolutional neural network. Representations of annotated images and a query image are each generated based on activations output by a layer of the model which precedes the fully-connected layers of the neural network. A similarity is computed between the query image representation and each of the annotated image representations to identify a subset of the annotated images having the highest computed similarity. Object location information from at least one of the subset of annotated images is transferred to the query image and information is output, based on the transferred object location information.

Claims

1. A method for object localization in a query image, comprising:

with a processor;

for each of a set of annotated images, with a model derived from part of a pre-trained neural network, the model including a sequence of convolutional layers, generating an annotated image representation based on activations output by a layer of the model, the annotated images each being annotated with object location information;

for a query image, generating a query image representation based on activations output by the layer of the model;

identifying a subset of similar annotated images from the set of annotated images, the identifying comprising computing a similarity between the query image representation and each of the annotated image representations;

transferring object location information from at least one of the subset of similar annotated images to the query image; and

outputting information based on the transferred object location information.

2. The method of claim 1, wherein the layer is preceded by at least three convolutional layers in the model.

3. The method of claim 1, wherein the layer of the model is a max-pooling layer.

4. The method of claim 3, wherein the max-pooling layer is the max-pooling layer immediately before a first fully-connected layer of the trained neural network.

5. The method of claim 1, wherein the similarity between the query image representation and each of the annotated image representations is computed as a cosine similarity between the representations.

6. The method of claim 1, where the similarity between the query image representation and each of the annotated image representations is computed is computed in a feature space by projecting the image representation and each of the annotated image representations with a learned metric.

7. The method of claim 6, further comprising learning the metric using a set of annotated training images to optimize a loss function.

8. The method of claim 6, further comprising jointly learning the metric and adapting the convolutional weights of the model by backpropagation.

9. The method of claim 8, comprising introducing a layer to the model after the last convolutional layer of the model, the layer serving as the metric.

10. The method of claim 1, wherein the output information comprises at least one of a bounding box for localizing an object in the query image and information extracted from the bounding box.

11. The method of claim 10, wherein the information extracted includes text extracted from a region of the image bounded by the bounding box.

12. The method of claim 11, wherein the information extracted comprises a license plate number.

13. The method of claim 1, wherein the providing of the annotated image representation comprises inputting the pixels of the annotated image into the model and outputting the activations of the layer of the model as a vector.

14. The method of claim 1, wherein the model comprises a sequence of convolutional layers which each receive as input output activations of a prior convolutional layer or the image in the case of a first convolutional layer.

15. The method of claim 1, further comprising ranking the annotated images based on the computed similarity and wherein the transferring object location information from at least one of the subset of annotated images to the query image comprises computing a weighted sum of bounding box annotations of the annotated images in the subset, the weights being based on the rankings of the annotated images in the subset.

16. A computer program product comprising a non-transitory recording medium storing instructions, which when executed on a computer, causes the computer to perform the method of claim 1.

17. A system comprising memory which stores instructions for performing the method of claim 1 and a processor in communication with the memory for executing the instructions.

18. A system for object localization in a query image, comprising:

memory which stores a model derived from a trained neural network, the model comprising a sequence of convolutional layers which receive as input output activations of a prior convolutional layer or an image in the case of a first convolutional layer;

a representation generator which generates a representation of a query image based on the activations of a selected one of the convolutional layers for the query image and an annotated image representation for each of a set of annotated images based on activations output by the selected layer of the model for the annotated image, the annotated images each being annotated with object location information;

a retrieval component which retrieves a subset of similar images from the set of annotated images, based on a similarity between respective representations;

a segmentation component which transfers the object location information from at least one of the subset of annotated images to the query image;

an output component which outputs information based on the transferred object location information; and

a processor which implements the representation generator, retrieval component, segmentation component, and output component.

19. The system of claim 18, further comprising a metric learning component which jointly learns a metric for projecting the image representations into a new feature space and adapts weights of the convolutional layers of the model by backpropagation.

20. A method for object localization in a query image, comprising:

with a processor;

adding a new layer to a pre-trained neural network to generate a model comprising a sequence of convolutional layers which each act on the output of a respective previous layer, the new layer being positioned after the last of the convolutional layers;

using a set of annotated training images, updating a matrix of weights of the new layer and weights of the convolutional layers of the model by backpropagation to optimize a loss function;

for each of a set of annotated images, generating an annotated image representation based on activations output by the new layer of the model, the annotated images each being annotated with object location information;

for a query image, generating a query image representation based on activations output by the new layer of the model;

identifying a subset of the annotated images comprising computing a similarity between the query image representation and each of the annotated image representations;

transferring object location information from at least one of the subset of annotated images to the query image; and

outputting information based on the transferred object location information.