Instance-level image retrieval with a region proposal network

• US 10,678,846 B2
• Filed: 03/10/2017
• Issued: 06/09/2020
• Est. Priority Date: 03/10/2017
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Gordo Soldevila, Albert, Almazan, Jon, Revaud, Jerome, Larlus-Larrondo, Diane
• Primary Examiner(s): Bezuayehu, Solomon G
• Application Number:
  US15/455,551
• Publication Number:
  US 20180260415A1
• Time in Patent Office: 1,187 Days
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

In a method for detecting an object in an input image, an input image vector representing the input image is generated by performing a regional maximum activations of convolutions (R-MAC) using a convolutional neural network (CNN) applied to the input image and using regions for the R-MAC defined by applying a region proposal network (RPN) to the output of the CNN applied to the input image. Likewise, a reference image vector representing a reference image depicting the object is generated by performing the R-MAC using the CNN applied to the reference image and using regions for the R MAC defined by applying the RPN to the output of the CNN applied to the reference image. A similarity metric between the input image vector and the reference image vector is computed, and the object is detected as present in the input image if the similarity metric satisfies a detection criterion.

Claims

1. An image processing device comprising:

an electronic processor; and

a non-transitory storage medium operatively connected with the electronic processor and storing instructions readable and executable by the electronic processor to perform a method for detecting an object in an input image by operations including;

generating an input image vector representing the input image by applying to the input image an image processing network including applying a convolutional neural network (CNN) to the input image to generate an input image CNN response map, defining regions of the input image CNN response map by applying a region proposal network (RPN) to the input image CNN response map, generating a region vector representing each region of the input image CNN response map, and sum-aggregating the region vectors representing the regions of the input image CNN response map;

for a reference image that is different from the input image and that depicts the object, generating a reference image vector representing the reference image by applying to the reference image the same image processing network as applied to the input image including applying the CNN to the reference image to generate a reference image CNN response map, defining regions of the reference image CNN response map by applying the RPN to the reference image CNN response map, generating a region vector representing each region of the reference image CNN response map, and sum-aggregating the region vectors representing the regions of the reference image CNN response map;

computing a similarity metric between the input image vector and the reference image vector; and

detecting the object in the input image if the similarity metric satisfies a detection criterion.

2. The image processing device of claim 1 wherein generating each region vector includes max pooling activation features in the corresponding region of the CNN response map.

3. The image processing device of claim 2 wherein generating each region vector further includes whitening the output of the max-pooling.

4. The image processing device of claim 1 wherein the region vectors representing the regions of the input image CNN response map, the region vectors representing the regions of the reference image CNN response map, the input image vector representing the input image, and the reference image vector representing the reference image are all of the same length.

5. The image processing device of claim 1 wherein the RPN comprises a fully convolutional network.

6. The image processing device of claim 1 wherein:

the generating of the input image vector representing the input image is performed as a single forward pass through the CNN and RPN and does not include any back-propagation; and

the generating of the reference image vector representing the reference image is performed as a single forward pass through the CNN and RPN and does not include any back-propagation.

7. The image processing device of claim 1 wherein the similarity metric is a dot product.

8. The image processing device of claim 1 wherein the non-transitory storage medium further stores instructions readable and executable by the electronic processor to perform a training method for training the CNN and the RPN, the training method including:

(i) training the CNN by operations including performing the method for detecting an object with the RPN replaced by fixed regions; and

(ii) training the RPN using the CNN trained in the operation (i).

9. The image processing device of claim 8 wherein the operation (i) employs a three-stream Siamese network that optimizes a triplet ranking loss for training the CNN.

10. The image processing device of claim 1 wherein the object is one of a person, an item, or a virtual reality object.

11. The image processing device of claim 1 wherein the non-transitory storage medium further stores instructions readable and executable by the electronic processor to perform a training method for training the CNN and the RPN, the training method including:

training the CNN by operations including performing the method for detecting an object with the RPN replaced by fixed regions by employing a network that optimizes a triplet ranking loss for training the CNN.

12. The image processing device of claim 1 wherein the non-transitory storage medium further stores instructions readable and executable by the electronic processor to perform a training method for training the CNN and the RPN, the training method including:

training the CNN by operations including performing the method for detecting an object with the RPN replaced by fixed regions by employing a network that maximizes a function with an input of a scalar that controls a margin.

13. The image processing device of claim 1 wherein the similarity metric is a Euclidean distance.

14. A method, performed by one or more processors, comprising:

generating an input image vector representing an input image by applying to the input image an image processing network including applying a convolutional neural network (CNN) to the input image to generate an input image CNN response map, defining regions of the input image CNN response map by applying a region proposal network (RPN) to the input image CNN response map, generating a region vector representing each region of the input image CNN response map, and sum-aggregating the region vectors representing the regions of the input image CNN response map;

for a reference image that is different from the input image and that depicts the object, generating a reference image vector representing the reference image by applying to the reference image the same image processing network as applied to the input image including applying the CNN to the reference image to generate a reference image CNN response map, defining regions of the reference image CNN response map by applying the RPN to the reference image CNN response map,

generating a region vector representing each region of the reference image CNN response map, and sum-aggregating the region vectors representing the regions of the reference image CNN response map;

computing a similarity metric between the input image vector and the reference image vector; and

detecting the object in the input image if the similarity metric satisfies a detection criterion.