INTERACTIVE CONCEPT LEARNING IN IMAGE SEARCH

• US 20090154795A1
• Filed: 12/12/2007
• Published: 06/18/2009
• Est. Priority Date: 12/12/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tan, Desney S., Winder, Simon A. J., Kapoor, Ashish, Fogarty, James A.
• Granted Patent: US 8,165,406 B2
• Time in Patent Office: Days
• US Class Current: 382/155
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures

Abstract

An interactive concept learning image search technique that allows end-users to quickly create their own rules for re-ranking images based on the image characteristics of the images. The image characteristics can include visual characteristics as well as semantic features or characteristics, or may include a combination of both. End-users can then rank or re-rank any current or future image search results according to their rule or rules. End-users provide examples of images each rule should match and examples of images the rule should reject. The technique learns the common image characteristics of the examples, and any current or future image search results can then be ranked or re-ranked according to the learned rules.

Claims

1. A computer-implemented process for finding one or more images of desired image characteristics, comprising:

obtaining a set of images;

providing example images with image characteristics sought to be found;

learning one or more rules for ranking images based on the image characteristics of the example images via a machine learning technique; and

ranking the set of images based on the one or more learned rules.

2. The computer-implemented process of claim 1 further comprising applying the one or more learned rules to a second set of images.

3. The computer-implemented process of claim 1 further comprising combining two or more learned rules before ranking the set of images.

4. The computer-implemented process of claim 1 further comprising obtaining the set of images in response to a search query submitted over a computer network.

5. The computer-implemented process of claim 1 further comprising obtaining the set of images by locally searching for images on a computing device.

6. The computer-implemented process of claim 1, further comprising repeating the actions of:

providing examples of images with image characteristics sought to be found;

learning one or more rules for ranking images based on the image characteristics of the example images via a machine learning technique; and

ranking the set of images based on the one or more learned rules until the images of the set of images are ranked with the images with image characteristics sought to be found being ranked highest.

7. The computer-implemented process of claim 1 wherein each rule is based on learning a distance metric which computes a score for each image returned in the set of images that indicates how similar each image returned in the set of images is to the examples of images with image characteristics sought to be found.

8. The computer-implemented process of claim 7, further comprising

computing a set of image histograms computed over the pixels in each image in the set of images;

computing two distance metrics for each histogram of the set of image histograms;

computing a structure histogram that represents the overall structure of each image in the set of images by applying a shape descriptor to the whole image;

computing a texture histogram for each image in the set of images; and

using the computed image histograms, the distance metrics for each of the computed image histograms, the structure histogram and the texture histogram for each image to compute a score for each image in the set of images that indicates how similar each image in the set of images is to the example images with image characteristics sought to be found and example images not having image characteristics sought to be found.

9. The computer-implemented process of claim 1 further comprising learning the one or more rules for ranking images based on a set of distance metrics that are used to represent the distance of any given image to any other given image.

10. The computer-implemented process of claim 8 wherein the two distance metrics comprise:

the quadratic distance between the image histograms of two images; and

the distance in histogram entropy between two images.

11. The computer-implemented process of claim 9 wherein the set of distance metrics are based on visual characteristics or semantic characteristics.

13. The computer-implemented process of claim 11 further comprising;

showing the ranked images to a user;

the user evaluating the ranked images and if not satisfied with the ranked images providing one or more example images until the user is satisfied that the most relevant images of the ranked images contain one or more images that the user is seeking.

12. A computer-implemented process for searching for an image of desired image characteristics, comprising:

obtaining a set of images;

computing visual features for each of the set of images;

selecting an example image, either a positive example image having features of a type of image being sought, or a negative example image not having features of the type of image being sought;

learning a rule by computing a weighting of features of each of the set of images based on one or more example images provided to determine the most relevant of the images of the image set; and

ranking the images of the image set according to the computed weighting.

14. The computer-implemented process of claim 12 further comprising providing the user with example images for the user to specify at least one image is more like the image the user is seeking prior to the user providing new example images.

15. The computer-implemented process of claim 12 further comprising applying one or more of the learned rules to a new set of images to rank the new set of images.

16. A system for ranking images retrieved in response to an image search, comprising:

a general purpose computing device;

a computer program comprising program modules executable by the general purpose computing device, wherein the computing device is directed by the program modules of the computer program to,

receive a set of images in response to an image search;

specify examples of images with image characteristics sought to be found;

extract features based on image characteristics of each of the set of images received in response to the image search and the specified example images;

learn a rule based on the features of the image characteristics of the example images;

weight the features of each of the images of the set of images;

rank the images of the image set based on one or more learned rules; and

display the ranked images of the image set to a user.

17. The system of claim 16 further comprising a module to apply one or more learned rules to rank a new set of images.

18. The system of claim 16 further comprising a module to use key words to retrieve the set of images in the image search.

19. The system of claim 18 further comprising using the key words combined with the features of the image characteristics sought to be found to learn one or more rules to rank the set of images.

20. The system of claim 16 further comprising a user interface which displays the ranked images of the image set and provides positive and negative example images for the user to choose to speed the learning of the rule.