Adaptive Visual Similarity for Text-Based Image Search Results Re-ranking

• US 20090313239A1
• Filed: 06/16/2008
• Published: 12/17/2009
• Est. Priority Date: 06/16/2008
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tang, Xiaoou, Wen, Fang
• Application Number:
  US12/140,244
• Publication Number:
  US 20090313239A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59

Abstract

Described is a technology in which images initially ranked by some relevance estimate (e.g., according to text-based similarities) are re-ranked according to visual similarity with a user-selected image. A user-selected image is received and classified into an intention class, such as a scenery class, portrait class, and so forth. The intention class is used to determine how visual features of other images compare with visual features of the user-selected image. For example, the comparing operation may use different feature weighting depending on which intention class was determined for the user-selected image. The other images are re-ranked based upon their computed similarity to the user-selected image, and returned as query results. Retuning of the feature weights using actual user-provided relevance feedback is also described.

Claims

1. In a computing environment, a method comprising:

receiving user selection data with respect to an image selected from a plurality of images, the selection data including a query image;

determining similarity scores for other images of the plurality based on each other image'"'"'s similarity with the query image, in which the similarity scores are computed at least in part based upon intention class information associated with the query image; and

returning results corresponding to the images ranked based upon the similarity scores.

2. The method of claim 1 wherein receiving the user selection data comprises receiving a user selection corresponding to the query image based upon text-ranked image results.

3. The method of claim 1 further comprising, classifying the query image into a class, and selecting the intention class information based on the class.

4. The method of claim 1 further comprising, featurizing the query image into first feature values and featurizing each other image into second feature values, and wherein determining the similarity scores comprises comparing data corresponding to the first and second feature values.

5. The method of claim 4 wherein comparing the data corresponding to the first and second feature values comprises weighing parts of the feature values relative to one another based upon the intention class information.

6. The method of claim 1 further comprising, tuning the intention class information based upon relevance feedback.

7. In a computing environment, a system comprising, an image processing mechanism, including a categorization mechanism that obtains an intention class for a selected image, a featurizer mechanism that obtains first feature values for the selected image and second feature values for another image, and a feature comparing mechanism coupled to the categorization mechanism and to the featurizer mechanism, the feature comparing mechanism configured to use the intention class to select a comparison mechanism, and use the comparison mechanism to compute a similarity score between the selected image and the other image using the first feature values and the second feature values.

8. The system of claim 7 wherein the selected image and the other image are provided by an Internet search engine coupled to the image processing mechanism.

9. The system of claim 7 wherein the image processing mechanism further includes a ranking mechanism that ranks the similarity score relative to at least one other similarity score obtained by processing another image.

10. The system of claim 7 further comprising a cache coupled to the image processing mechanism, wherein the featurizer mechanism obtains at least some of the first feature values, or at least some of the second feature values, or at least some of both the first feature values and the second feature values from the cache.

11. The system of claim 7 further comprising a cache coupled to the image processing mechanism, wherein the categorization mechanism obtains the intention class from the cache.

12. The system of claim 7 further comprising means for tuning the comparison mechanism based upon relevance feedback.

13. The system of claim 11 wherein the comparison mechanism comprises a set of feature weights selected from among a plurality of sets of feature weights.

14. The system of claim 13 wherein the features include color signature, color spatialet, gist, Daubechies wavelet, SIFT, multi-layer rotation invariant edge orientation histogram, histogram of gradient, or facial feature face, or any combination of color signature, color spatialet, gist, Daubechies wavelet, SIFT, multi-layer rotation invariant edge orientation histogram, histogram of gradient, or facial feature face.

15. The system of claim 13 wherein the classes include general object, simple background object, scene, people, portrait or other, or any combination of general object, simple background object, scene, people, portrait or other.

16. One or more computer-readable media having computer-executable instructions, which when executed perform steps, comprising:

(a) receiving data corresponding to a set of images and one selected image;

(b) classifying the selected image into an intention class;

(c) choosing a comparison mechanism from among plurality of available comparison mechanisms based upon the intention class;

(d) featurizing the selected image into first feature values;

(e) for each image other than the selected image, taking that image as a comparison image, featurizing that comparison image into second feature values, and comparing the first feature values and the second feature values using the comparison mechanism chosen in step (c) to determine and associate a similarity score of the comparison image with respect to that comparison image; and

(f) returning data corresponding to the comparison images re-ranked relative to one another based on the associated similarity score determined for each image.

17. The one or more computer-readable media of claim 16 wherein choosing the comparison mechanism comprises selecting a set of feature weights from among different sets of feature weights based upon the intention class.

18. The one or more computer-readable media of claim 16 having further computer-executable instructions comprising, changing at least one comparison mechanism based upon user relevance feedback.

19. The one or more computer-readable media of claim 16, wherein the features include color signature, color spatialet, gist, Daubechies wavelet, SIFT, multi-layer rotation invariant edge orientation histogram, histogram of gradient, or facial feature face, or any combination of color signature, color spatialet, gist, Daubechies wavelet, SIFT, multi-layer rotation invariant edge orientation histogram, histogram of gradient, or facial feature face.

20. The one or more computer-readable media of claim 16, wherein the classes include general object, simple background object, scene, people, portrait or other, or any combination of general object, simple background object, scene, people, portrait or other.