Image retrieval systems and methods with semantic and feature based relevance feedback

• US 7,099,860 B1
• Filed: 10/30/2000
• Issued: 08/29/2006
• Est. Priority Date: 10/30/2000
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lu, Ye, Zhang, Hong-Jiang, Liu, Wen-Yin
• Primary Examiner(s): Corrielus, Jean M.
• Assistant Examiner(s): To, Baoquoc N
• Application Number:
  US09/702,292
• Time in Patent Office: 2,129 Days
• Field of Search: 707/3, 707/4, 707/5, 707/10, 707/103
• US Class Current: 1/1
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

An image retrieval system performs both keyword-based and content-based image retrieval. A user interface allows a user to specify queries using a combination of keywords and examples images. Depending on the input query, the image retrieval system finds images with keywords that match the keywords in the query and/or images with similar low-level features, such as color, texture, and shape. The system ranks the images and returns them to the user. The user interface allows the user to identify images that are more relevant to the query, as well as images that are less or not relevant to the query. The user may alternatively elect to refine the search by selecting one example image from the result set and submitting its low-level features in a new query. The image retrieval system monitors the user feedback and uses it to refine any search efforts and to train itself for future search queries. In the described implementation, the image retrieval system seamlessly integrates feature-based relevance feedback and semantic-based relevance feedback.

Claims

1. A method comprising:

initiating a search for images based on at least one query keyword in a query;

identifying, during the search, first images having associated keywords that match the query keyword, and second images that contain low-level features similar to those of the first images identified as matching the query keyword, wherein the low-levels features comprise color, shape, and texture and the low-level features do not match the search query;

ranking the first and second images;

presenting the first and second images to a user; and

receiving feedback from the user as to whether the first and second images are relevant to the query further comprising;

assigning a large weight to an association between the query keyword and the images deemed relevant by the user and/or assigning a small weight to an association between the query keyword and the example image.

2. A method as recited in claim 1, further comprising presenting the first and second images.

5. A method as recited in claim 1, wherein the monitoring comprises

presenting the first and second images to a user;

receiving feedback from the user as to whether the first and second images are relevant to the query further comprising;

refining the search to identify additional images that contain low-level features similar to those of the images indicated by the user as being relevant to the a query.

6. A method as recited in claim 1, wherein the monitoring comprises

presenting the first and second images to a user,

receiving feedback from the user as to whether the first and second images are relevant to the query further comprising:

assigning a large weight to an association between the query keyword and the images deemed relevant by the user.

7. A method as recited in claim 6, further comprising grouping the low-level features of the images deemed relevant by the user.

8. A method as recited in claim 1, wherein the monitoring comprises

presenting the first and second images to a user;

receiving feedback from the user as to whether the first and second images are relevant to the query further comprising;

assigning a small weight to an association between the query keyword and the example image.

9. A method as recited in claims 8, further comprising identifying additional images with low-level features similar to those of the example image.

10. A computer readable medium having computer-executable instructions that, when executed on a processor, perform the method as recited in claim 1.

3. A method comprising:

initiating a search for images based on at least one query keyword in a query;

identifying, during the search, first images having associated keywords that match the query keyword, and second images that contain low-level features similar to those of the first images identified as matching the query keyword wherein the low-levels features comprise color, shape and texture and the low-level features do not match the search query;

presenting the first and second images to a user;

monitoring feedback from the user as to which of the first and second images are relevant to the query; and

receiving feedback from the user as to whether the first and second images are relevant to the query comprising;

assigning a large weight to an association between the query keyword and the images deemed relevant by the user and/or assigning a small weight to an association between the query keyword and the example image.

31. A computer readable medium having computer-executable instructions that, when executed on a processor, perform the method as recited in claim 3.

4. A method comprising:

initiating a search for images based on at least one query keyword in a query;

identifying, during the search, first images having associated keywords that match the query keyword, and second images that contain low-level features similar to those of the first images identified as matching the query keyword wherein the low-levels features comprise color, shape, and size and the low-level features do not match the search query;

presenting the first and second images to a user;

receiving feedback from the user as to whether the first and second images are relevant to the query;

learning how the first and second images are identified based on the feedback from the user; and

receiving feedback from the user as to whether the first and second images are relevant to the query further comprising;

assigning a large, weight to an association between the query keyword and the images deemed relevant by the user and/or assigning a small weight to an association between the query keyword and the example image.

11. A method comprising:

permitting entry of both keyword-based queries and content-based queries;

finding a first set of images using semantic-based image retrieval and a second set of images using low-level feature-based image retrieval based on the first set of images wherein the low-levels features comprise color, shape and texture and the low-level features do not match the semantic-based and the low-level features do not match the keyword-based or content-based;

presenting the images to a user so that the user can indicate whether the images are relevant;

conducting semantic-based relevance feedback and low-level feature-based relevance feedback in an integrated fashion; and

receiving feedback from the user as to whether the first and second images are relevant to the query further comprising;

assigning a large weight to an association between the query keyword and the images deemed relevant by the user and/or assigning a small weight to an association between the query keyword and the example image.

12. A method as recited in claim 11, further comprising ranking images.

13. A method as recited in claim 11, further comprising using images indicated as being relevant to find additional images.

14. A computer readable medium having computer-executable instructions that, when executed on a processor, perform the method as recited in claim 11.

15. A method comprising:

identifying the set of images from the presenting retrieved using the at least one keyword and second images that contain low-level features similar to those of the first images identified as matching the at least one keyword, wherein the low-levels features comprise color, shape and texture and the low-level features do not match the search query;

presenting a result set of images that are returned from an image retrieval search of a query having at least one keyword;

monitoring feedback from a user as to whether the images in the result set are relevant to the query;

in an event that the user selects at least one image as being relevant to the query, associating the keyword in the query with the selected image to form a first keyword-image association and assigning a comparatively large weight to the first keyword-image association; and

in an event that the user identifies an example image for refinement of the search, associating the keyword in the query with the example image to form a second keyword-image association and assigning a comparatively small weight to the second keyword-image association.

16. A method as recited in claim 15, further comprising conducting both content-based image retrieval and semantic-based image retrieval.

17. A method as recited in claim 15, further comprising presenting the result set of images in a user interface, the user interface facilitating the user feedback by allowing the user to indicate which images are more relevant and which images are less relevant.

18. A computer readable medium having computer-executable instructions that, when executed on a processor, perform the method as recited in claim 15.

19. An image retrieval system comprising:

a query handler to handle both keyword-based queries having one or more search keywords and content-based queries having one or more low-level features of an image; and

a feature and semantic matcher to identify at least one of (1) first images having keywords that match the search keywords from a keyword-based query, and (2) second images having low-level features similar to the low-level features of a content-based query based on the first images, wherein the feature and semantic matcher ranks the images, and the low-level features comprise color, shape and texture, and the low-level features do not match the keyword-based or content-based;

presenting the first and second images to a user; and

receiving feedback from the user as to whether the first and second images are relevant to the query further comprising;

assigning a large weight to an association between the query keyword and the images deemed relevant by the user and/or assigning a small weight to an association between the query keyword and the example image.

20. An image retrieval system as recited in claim 19, wherein the query handler comprises a natural language parser.

21. An image retrieval system as recited in claim 19, wherein query handler comprises:

a parser to parse textbased queries; and

a concept hiearchy to define various categories of images.

22. An image retrieval system as recited in claim 19, further comprising a user interface to present the images identified by the feature and semantic matcher.

24. An image retrieval system as recited in claim 19, further comprising:

a user interface to present the images identified by the feature and semantic matcher to a user, the user interface allowing the user to identify an example image; and

the feature and semantic matcher being configured to identify additional images that contain low-level features similar to those of the example image.

25. An image retrieval system as recited in claim 19, further comprising:

a user interface to present the images identified by the feature and semantic matcher to a user, the user interface allowing the user to identify which images are relevant to a particular search query; and

a feedback analyzer to assign a large weight to an association between the search keywords and the images identified as relevant by the user.

26. An image retrieval system as recited in claim 25, wherein the feedback analyzer groups the low-level features of the images identified as relevant by the user.

23. An image retrieval system comprising:

a query handler to handle both keyword-based queries having one or more search keywords and content-based queries having one or more low-level features of an image;

a feature and semantic matcher to identify at least one of (1) first images having keywords that match the search keywords from a keyword-based query, and (2) second images having low-level features similar to the low-level features of a content-based query based on the first images, wherein the low-level features comprise color, shape and texture and the low-level features do not match the keyword-based or the content-based;

a user interface to present the images identified by the feature and semantic matcher to a user, the user interface allowing the user to indicate whether the images are relevant to the query; and

a feedback analyzer to train the image retrieval system based on user feedback as to relevancy, wherein the feedback analyzer to assign a large weight to an association between the query keyword and the images deemed relevant by the user and/or feedback analyzer to assign a small weight to an association between the query keyword and the example image and/or a feedback analyzer to assign a small weight to an association between the search keywords and the example image.

27. An image retrieval system as recited in claim 23, wherein the feature and semantic matcher identifies additional images with low-level features similar to those of the example image.

28. A computer-readable medium having computer-executable instructions that, when executed, directs a computer to:

find a first set of images using both semantic-based image retrieval, and a second set of images using low-level feature-based image retrieval based on the first set of images, wherein the low-level features comprise color, shape and texture and the low-level features do not match the semantic-based or feature-based;

present the images to a user so that the user can indicate whether the images are relevant; and

concurrently conduct semantic-based relevance feedback and low-level feature-based relevance feedback,

wherein the receiving feedback from the user as to whether the first and second images are relevant to the query further comprising;

assigning a large weight to an association between the query keyword and the images deemed relevant by the user and/or assigning a small weight to an association between the query keyword and the example image.

29. A program as recited in claim 28, further comprising computer-executable instructions that, when executed, direct a computer to rank the images.

30. An information retrieval program, embodied on the computer-readable medium, comprising the computer-executable instructions of claim 28.