Semi-automatic annotation of multimedia objects

• US 20050010553A1
• Filed: 07/28/2004
• Published: 01/13/2005
• Est. Priority Date: 10/30/2000
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Liu, Wen-Yin
• Granted Patent: US 7,627,556 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation

Abstract

A multimedia object retrieval and annotation system integrates an annotation process with object retrieval and relevance feedback processes. The annotation process annotates multimedia objects, such as digital images, with semantically relevant keywords. The annotation process is performed in background, hidden from the user, as the user conducts normal searches. The annotation process is “semi-automatic” in that it utilizes both keyword-based information retrieval and content-based image retrieval techniques to automatically search for multimedia objects, and then encourages users to provide feedback on the retrieved objects. The user identifies objects as either relevant or irrelevant to the query keywords and based on this feedback, the system automatically annotates the objects with semantically relevant keywords and/or updates associations between the keywords and objects. As the retrieval-feedback-annotation cycle is repeated, the annotation coverage and accuracy of future searches continues to improve.

Claims

1. A method comprising:

maintaining associations between keywords and multimedia objects, the associations being weighted to indicate how relevant the keywords are to the multimedia objects;

retrieving a set of one or more multimedia objects for presentation to a user;

monitoring feedback from the user as to which of the multimedia objects are relevant; and

adjusting the weights of the associations based on the user'"'"'s feedback.

2. A method as recited in claim 1, wherein the retrieving comprises using content-based information retrieval to retrieve the multimedia objects.

3. A method as recited in claim 1, wherein the retrieving comprises using both content-based information retrieval and semantic-based information retrieval to retrieve the multimedia objects.

4. A method as recited in claim 1, wherein the monitoring comprises capturing both feature-based relevance feedback and semantic-based relevance feedback.

5. A method as recited in claim 1, wherein the adjusting comprises increasing the weights of the associations between the keywords and the multimedia objects that are deemed relevant by the user.

6. A method as recited in claim 1, wherein the adjusting comprises decreasing the weights of the associations between the keywords and the multimedia objects that are deemed irrelevant by the user.

7. A computer readable medium having computer-executable instructions that, when executed on a processor, perform the method as recited in claim 1.

8. A user interface, comprising:

a query interface to accept a search query for searching multimedia objects in a database system;

a browser to permit a user to browse the multimedia objects returned from the database system; and

a feedback interface to enable the user to indicate which multimedia objects are relevant to the search queries.

9. A user interface as recited in claim 8, wherein the query interface is configured to allow entry of both keyword-based queries with one or more keywords and content-based queries based on selection of an example multimedia object.