Semi-automatic annotation of multimedia objects

• US 7,627,556 B2
• Filed: 07/28/2004
• Issued: 12/01/2009
• Est. Priority Date: 10/30/2000
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Hong-Jiang, Liu, Wen-Yin
• Primary Examiner(s): To; Baoquoc N
• Application Number:
  US10/900,951
• Publication Number:
  US 20050010553A1
• Time in Patent Office: 1,952 Days
• Field of Search: 707/3, 707/4, 707/10, 707/100
• US Class Current: 1/1
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation

Abstract

A multimedia object retrieval and annotation system integrates an annotation process with object retrieval and relevance feedback processes. The annotation process annotates multimedia objects, such as digital images, with semantically relevant keywords. The annotation process is performed in background, hidden from the user, as the user conducts normal searches. The annotation process is “semi-automatic” in that it utilizes both keyword-based information retrieval and content-based image retrieval techniques to automatically search for multimedia objects, and then encourages users to provide feedback on the retrieved objects. The user identifies objects as either relevant or irrelevant to the query keywords and based on this feedback, the system automatically annotates the objects with semantically relevant keywords and/or updates associations between the keywords and objects. As the retrieval-feedback-annotation cycle is repeated, the annotation coverage and accuracy of future searches continues to improve.

Claims

1. A computer-implemented method comprising:

storing on a computer readable memory processor-executable instructions;

processing on a processor the processor-executable instructions, wherein the processor-executable instructions perform a method comprising;

maintaining association links between keywords and multimedia objects, the association links being weighted to indicate how relevant the keywords are to the multimedia objects;

retrieving a set of one or more multimedia objects for presentation to a user;

monitoring feedback from the user as to which of the multimedia objects are relevant; and

adjusting the weights of the association of the links between the keywords and multimedia objects, based on the user'"'"'s feedback, wherein the associations are made available for image retrieval and the adjusting comprises increasing and decreasing the weights of the associations between the keywords and the multimedia objects that are deemed relevant by the user, wherein relevance is determined via user feedback.

2. A method as recited in claim 1, wherein the retrieving comprises using content-based information retrieval to retrieve the multimedia objects.

3. A method as recited in claim 1, wherein the retrieving comprises using both content-based information retrieval and semantic-based information retrieval to retrieve the multimedia objects.

4. A method as recited in claim 1, wherein the monitoring comprises capturing both feature-based relevance feedback and semantic-based relevance feedback.

5. A computer readable memory having computer-executable instructions that, when executed on a processor, perform the method as recited in claim 1.