Image retrieval systems and methods with semantic and feature based relevance feedback

• US 20040267740A1
• Filed: 07/28/2004
• Published: 12/30/2004
• Est. Priority Date: 10/30/2000
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Hong-Jiang, Lu, Ye, Liu, Wen-Yin
• Granted Patent: US 7,529,732 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

An image retrieval system performs both keyword-based and content-based image retrieval. A user interface allows a user to specify queries using a combination of keywords and examples images. Depending on the input query, the image retrieval system finds images with keywords that match the keywords in the query and/or images with similar low-level features, such as color, texture, and shape. The system ranks the images and returns them to the user. The user interface allows the user to identify images that are more relevant to the query, as well as images that are less or not relevant to the query. The user may alternatively elect to refine the search by selecting one example image from the result set and submitting its low-level features in a new query. The image retrieval system monitors the user feedback and uses it to refine any search efforts and to train itself for future search queries. In the described implementation, the image retrieval system seamlessly integrates feature-based relevance feedback and semantic-based relevance feedback.

Claims

1. A method comprising:

initiating a search for images based on at least one query keyword in a query; and

identifying, during the search, first images having associated keywords that match the query keyword and second images that contain low-level features similar to those of the first images.

2. A method comprising:

associating keywords with images to form keyword-image links;

assigning weights to the keyword-image links;

presenting a result set of images obtained from an image retrieval search based on a query;

receiving feedback from a user as to whether the images in the result set are relevant to the query; and

modifying the weights according to the user feedback.

3. A method as recited in claim 2, wherein the modifying comprises increasing the weight of a keyword-image link for images deemed by the user as more relevant to the query.

4. A method as recited in claim 2, wherein the modifying comprises decreasing the weight of a keyword-image link for images deemed by the user as less relevant to the query.

5. A computer readable medium having computer-executable instructions that, when executed on a processor, perform the method as recited in claim 2.

6. A method comprising:

computing, for each category, a representative feature vectors of a set of existing images within the category;

determining a set of representative keywords that are associated with the existing images in each category;

comparing, for each new image, the low-level feature vectors of the new image to the representative feature vectors of the existing images in each category to identify a closest matching category; and

labeling the new image with the set of representative keywords associated with the closest matching category.

7. A method as recited in claim 6, further comprising using user feedback to selectively add and/or remove keywords from the new image.

8. A method as recited in claim 6, further comprising:

placing the labeled new images into a holding category;

evaluating the labeled new images in the holding category to determine if any of the keywords associated with the labeled new image match the representative keywords from each category; and

assigning the labeled new image to the category that best matches the keywords associated with the labeled new image.

9. An image retrieval system comprising:

a query handler to handle both keyword-based queries having one or more search keywords and content-based queries having one or more low-level features of an image; and

a feature and semantic matcher to identify at least one of (1) first images having keywords that match the search keywords from a keyword-based query, and (2) second images having low-level features similar to the low-level features of a content-based query.

10. A database structure stored on one or more computer-readable media comprising:

multiple image files;

multiple keywords; and

a semantic network to associate the keywords with the image files, the semantic network defining individual keyword-image links that associate a particular keyword with a particular image file, each keyword-image link having a weight indicative of how relevant the particular keyword is to the particular image file.