Image retrieval using discriminative visual features

• US 9,229,956 B2
• Filed: 01/10/2011
• Issued: 01/05/2016
• Est. Priority Date: 01/10/2011
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ke, Qifa, Deng, Jia, Baker, Simon, Isard, Michael
• Primary Examiner(s): Lefkowitz, Sumati
• Assistant Examiner(s): Wang, Carol
• Application Number:
  US12/987,167
• Publication Number:
  US 20120177294A1
• Time in Patent Office: 1,821 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship

Abstract

Image search results are obtained by providing weights to visual features to emphasize features corresponding to objects of interest while simultaneously deemphasizing irrelevant or inconsistent features that lead to poor search results. In order to minimize the impact of visual features that are unreliable or irrelevant with respect to the objects of interest in the image, context-dependent weights are provided to detect visual features such that those visual features pertaining to the objects of interest are more heavily weighted than those visual features that pertain to irrelevant or unreliable portions of the image. Visual features may be weighted for images in a searchable database. Training data may be obtained and used in weighting visual features in a query image and, alternatively, in searchable database images.

Claims

1. A method of weighting visual features in an image, the method comprising:

detecting, by a computing device, a plurality of visual features for the image;

determining, by the computing device, a plurality of descriptors based on the plurality of visual features;

determining, by the computing device, a visual word corresponding to each of the plurality of descriptors by quantizing each descriptor using a collection of known visual words;

determining, by the computing device, a weight for each of the plurality of descriptors, wherein determining a weight for a descriptor comprises calculating the weight based on a vector associated with the visual word corresponding to the descriptor and a concatenated vector of a local context and a global context and based on a difference between the vector associated with the visual word corresponding to the descriptor and vectors associated with visual words corresponding to one or more descriptors of the plurality of descriptors that are neighbors of the descriptor; and

adding the image to a database of images by the computing device, wherein the database of images is indexed based on the weighted descriptors.

2. The method of claim 1, wherein the image comprises a query image or a database image.

3. The method of claim 1, further comprising computationally favoring at least one higher-weighted descriptor over a lower-weighted descriptor in an image comparison operation.

4. The method of claim 1, further comprising determining a threshold weight, and eliminating at least one descriptor having a weight below the threshold.

5. The method of claim 1, wherein a weight assigned to at least one visual feature from among the plurality of visual features is derived from a predefined weighting for a visual word corresponding to the visual feature.

6. The method of claim 1, wherein a weight is further calculated using at least one global statistic from the image.

7. The method of claim 1, wherein a weight is further calculated using at least one local statistic from a region corresponding to a visual feature from among the plurality of visual features in the image to which the weight pertains.

8. The method of claim 1, wherein a weight is further calculated using at least one geometric relationship based on a visual feature from among the plurality of visual features in the image to which the weight pertains.

9. The method of claim 1, wherein a weight is learned from an optimization process using at least one triplet of images comprising a query image, a relevant image, and an irrelevant image.

10. The method of claim 1, further comprising determining a histogram of the determined visual words.

11. The method of claim 1, further comprising reducing a dimensionality of the concatenated vector prior to calculating the weight based on the vector associated with the visual word corresponding to the descriptor and the concatenated vector.

12. The method of claim 1, wherein the vector associated with the visual word is a feature vector, and further comprising clustering the visual words into a plurality of clusters based on the feature vectors associated with the visual words, and for each cluster of the plurality of clusters, associating a same feature vector to each visual word in the cluster.

13. A method of searching for images in a corpus of searchable images based on a query image, the method comprising:

detecting, by a computing device, a plurality of visual features for the query image;

computing, by the computing device, a plurality of descriptors based on the plurality of visual features;

computing, by the computing device, a visual word corresponding to each of the plurality of descriptors by quantizing each descriptor using a collection of known visual words;

computing, by the computing device, a weight for each descriptor among the plurality of descriptors to produce a plurality of weighted descriptors, wherein computing a weight for a descriptor comprises calculating the weight based on a vector associated with the visual word corresponding to the descriptor and a concatenated vector of a local context and a global context and based on a difference between the vector associated with the visual word corresponding to the descriptor and vectors associated with visual words corresponding to one or more descriptors of the plurality of descriptors that are neighbors of the descriptor;

utilizing the plurality of descriptors to retrieve a plurality of candidate images from the corpus of searchable images based on the query image, by the computing device, wherein the images of the corpus of searchable images are indexed by the weighted descriptors;

ranking the plurality of candidate images using spatial verification; and

sorting and presenting the ranked plurality of candidate images based on a strength of the match between each candidate image and the query image by the computing device.

14. The method of claim 13, further comprising eliminating a subset of descriptors from among the plurality of descriptors based on the weights of the descriptors in the subset of descriptors.

15. The method of claim 13, wherein the weight is further calculated using a basic weighting approach based on a constant weight given for a corresponding normalized visual word.

16. An image search system comprising a computing device, the computing device comprising:

a feature detection component for detecting at least one visual feature in an image;

a feature description component for abstracting from the at least one visual feature a plurality of vectors pertaining to at least one vector abstraction from among the following vector abstractions;

intensity, rotation, and scale;

a visual word conversion component for quantizing the at least one resulting vector into at least one visual word using a collection of known visual words;

a weighting engine for weighting the at least one visual word by calculating a weight based on a vector associated with the visual word and a concatenated vector of a local context and a global context and based on a difference between the vector associated with the visual word and vectors associated with visual words that are neighbors of the at least one visual word; and

an image store for storing the image, wherein a plurality of images in the image store are indexed based on the weighted visual words.

17. The system of claim 16, further comprising a spatial and geometric relationship engine for quantifying the spatial relationships between at least two visual features extracted from the image.

18. The system of claim 16, further comprising a similarity engine for comparing a query image to a retrieved database image to assess the level of similarity between the query image and the retrieved database image.

19. The system of claim 16, further comprising a learning engine for obtaining training data that may be used in weighting a plurality of visual features in the image.

20. The system of claim 19, wherein the learning engine uses an optimization process using at least one triplet of images comprising a query image, a relevant image, and an irrelevant image.