Annotation by search

• US 8,341,112 B2
• Filed: 05/19/2006
• Issued: 12/25/2012
• Est. Priority Date: 05/19/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Lei, Jing, Feng, Ma, Wei-Ying, Wang, Xin-ing
• Primary Examiner(s): Pwu, Jeffrey
• Assistant Examiner(s): SONG, HEE K
• Application Number:
  US11/419,368
• Publication Number:
  US 20070271226A1
• Time in Patent Office: 2,412 Days
• Field of Search: 713/180, 707 1- 7, 707/100
• US Class Current: 707/602
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

Annotation by search is described. In one aspect, a data store is searched for images that are semantically related to a baseline annotation of a given image and visually similar to the given image. The given image is then annotated with common concepts of annotations associated with at least a subset of the semantically and visually related images.

Claims

1. A memory storing computer program instructions executable by a processor, the computer program instructions including instructions for performing operations comprising:

searching one or more data stores for images that are semantically related to a baseline annotation of a given image and visually similar to the given image;

measuring visual similarity of the semantically related images to the given image, wherein at least a subset of the images are selected from certain ones of the images determined to be more visually similar to the given image than other ones of the images;

calculating properties for phrases extracted from respective ones of annotations associated with the at least a subset of the images, the annotations being grouped into a respective cluster of multiple clusters;

combining, for each cluster, associated ones of the properties to generate a salience score for the cluster, wherein common concepts are associated with one or more clusters having higher salience scores as compared to salience scores associated with other ones of the clusters; and

annotating the given image with common concepts of annotations associated with at least a subset of the images that are both semantically related and visually similar to the given image.

2. The memory of claim 1, wherein the computer-program instructions further comprise instructions for removing duplicate-terms from the common concepts prior to annotating the given image.

3. The memory of claim 1, wherein the computer-program instructions further comprise instructions for receiving a search query comprising the given image and the baseline annotation.

4. The memory of claim 1, wherein the computer program instructions further comprise instructions for:

retrieving the images that are semantically related from the one or more data stores being searched; and

determining a common concept of annotations based at least on one or more weighted clusters of annotations associated with at least a subset of the images being retrieved, wherein the annotating includes annotating the given image with the common concept of annotations.

5. A computer-implemented method comprising:

searching for a first set of images that are semantically related to a baseline annotation of a given image;

evaluating each image in the first set of images to identify a second set of images that are at least conceptually related in visual space to the given image;

identifying annotations associated with the second set of images;

determining concepts common across respective ones of the annotations wherein determining concepts comprises;

clustering the annotations into a plurality of clusters such that each cluster of the clusters is associated with a salient concept of the cluster;

determining salience scores for the each salient concept of multiple salient concepts associated with the clusters;

selecting a set of top ranked salience concepts from the multiple salient concepts based on respective ones of the salience scores, wherein at least a subset of the top ranked salience concepts are the complimentary annotations; and

annotating the given image with at least a subset of the concepts as complimentary annotations.

6. The method of claim 5, further comprising receiving input comprising the given image and the baseline annotation.

7. The method of claim 5, wherein the searching further comprises correlating the baseline annotation with texts associated with the first set of images.

8. The method of claim 5, wherein the evaluating further comprises comparing hash signature encoded visual features of the given image with respective hash signature encoded visual features of each image in the first set of images.

9. The method of claim 5, wherein the evaluating further comprises:

measuring visual similarity of each image in the first set of images to the given image;

identifying, based on visual similarity measurements determined in the measuring, a set of top ranked images that are more visually related to the given image than other images from the first set of images; and

wherein the second set of images are the top ranked images.

10. The method of claim 5, wherein identifying annotations further comprises extracting text from contexts of the second set of images.

11. A computing device comprising:

a processor; and

a memory coupled to the processor, the memory comprising computer-program instructions that, when executed by the processor, perform operations of;

receiving a search query comprising a given image and a baseline annotation of the given image;

searching for a first set of images that are semantically related to the baseline annotation of the given image;

evaluating each image in the first set of images to identify a second set of images that are visually related to the given image, wherein the second set of images comprise certain images determined to be more visually related to the given image than other ones of the images;

calculating properties for phrases extracted from respective ones of the annotations, the annotations being grouped into a respective cluster of multiple clusters;

combining, for each cluster, associated ones of the properties to generate a salience score for the cluster, wherein common concepts are associated with one or more clusters having higher salience scores as compared to salience scores associated with other ones of the clusters; and

supplementing the given image with additional annotations associated with the second set of images, each image in the first second set of images being visually similar to the given image and associated with annotations that are semantically related to the baseline annotation, wherein the additional annotations are selected from image annotations and not from using a predetermined list.

12. The computing device of claim 11, wherein the additional annotations represent one or more of a title, a description, a category, information from a web page, a folder name, at least a portion of a Uniform Resource Identifier, and text in association with a different image that is not in the first set of images.

13. The computing device of claim 11, wherein the computer-program instructions further comprise instructions for deriving the first set of images by mining the images from one or more image databases, each image in the first set of images being mined based on text associated with the given image.

14. The computing device of claim 11, wherein the computer-program instructions further comprise instructions for:

performing a keyword based search to locate the first set of images that are semantically related to the baseline annotation;

searching the first set of images for one or more images that have substantial visual similarity to the given image;

extracting text from one or more contexts of the one or more images that have substantial visual similarity to the given image; and

wherein the text comprises at least additional annotations.

15. The computing device of claim 14, wherein the computer-program instructions further comprise instructions for:

identifying phrases in the text;

assigning salience scores to the phrases based on respective phrase properties; and

wherein the additional annotations comprise at least a portion of the text with higher salience scores as compared to salience scores associated with other portions of the text.

16. The computing device of claim 15, wherein the respective phrase properties comprise one or more of phrase frequencies and phrase-to-annotation frequencies.