Image retrieval systems and methods with semantic and feature based relevance feedback

• US 7,529,732 B2
• Filed: 07/28/2004
• Issued: 05/05/2009
• Est. Priority Date: 10/30/2000
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Hong-Jiang, Lu, Ye, Liu, Wen-Yin
• Primary Examiner(s): To; Baoquoc N
• Application Number:
  US10/900,574
• Publication Number:
  US 20040267740A1
• Time in Patent Office: 1,742 Days
• Field of Search: 707/3, 707/10, 707/4, 707/2
• US Class Current: 1/1
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

An image retrieval system performs both keyword-based and content-based image retrieval. A user interface allows a user to specify queries using a combination of keywords and examples images. Depending on the input query, the image retrieval system finds images with keywords that match the keywords in the query and/or images with similar low-level features, such as color, texture, and shape. The system ranks the images and returns them to the user. The user interface allows the user to identify images that are more relevant to the query, as well as images that are less or not relevant to the query. The user may alternatively elect to refine the search by selecting one example image from the result set and submitting its low-level features in a new query. The image retrieval system monitors the user feedback and uses it to refine any search efforts and to train itself for future search queries. In the described implementation, the image retrieval system seamlessly integrates feature-based relevance feedback and semantic-based relevance feedback.

Claims

1. A method executable by computing device configured for image retrieval, the method comprising:

Initiating a search for first images based on query keywords in a query;

Identifying, during the search, the first images having associated keywords that match the query keyword, and second images that contain low-level features similar to those of the first images, wherein the second images are retrieved based on the low-level features;

Receiving semantic-based relevance feedback and low-level feature relevance feedback;

Wherein their semantic-based relevance feedback includes feedback for strengthening or weakening associations between keywords of a search query and the retrieved images, wherein the low-level feature relevance feedback includes feedback for ranking the relevance of each of the retrieved images, wherein a basis of the low-level feature relevance feedback includes one or more of a color histogram, a texture, or a shape in the retrieved images;

Implementing a machine learning algorithm based on the received semantic-based relevance feedback and low-level relevance feedback; and

Storing the results of the search, the semantic-based relevance feedback and the low-level feature relevance feedback, for later use.

2. An image retrieval system comprising:

A query handler to handle both keyword-based queries having one or more search keywords and content-based queries having one or more low-level features of an image;

A feature and semantic matcher to identify (1) first images having keywords that match the search keywords from a keyword-based query, and (2) query images having low-level features similar to the low-level features of a content-based query, wherein the second images are retrieved based on the low-level features;

A feedback analyzer to monitor relevance feedback and implement a machine learning algorithm to improve image retrieval, wherein the relevance feedback includes feedback for strengthening or weakening associations between keywords of a search query and the retrieved images, and feedback for ranking the relevance of each of the retrieved images being based on one or more of a color histogram, a texture, or a shape in the retrieved images; and

A memory to store results of the search, the semantic-based relevance feedback and the low-level feature relevance feedback, for later use.

3. The image retrieval system of claim 2 further comprising a user interface to receive relevance feedback, wherein the relevance feedback includes indications on an image-by-image basis that the retrieved images are one or more of the following:

more relevant, less relevant or irrelevant.