System and method for semantically annotating images

• US 9,239,848 B2
• Filed: 02/06/2012
• Issued: 01/19/2016
• Est. Priority Date: 02/06/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Liu, Ce, Rubinstein, Michael
• Primary Examiner(s): Krasnic, Bernard
• Application Number:
  US13/367,139
• Publication Number:
  US 20130202205A1
• Time in Patent Office: 1,443 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information

Abstract

Techniques for semantically annotating images in a plurality of images, each image in the plurality of images comprising at least one image region. The techniques include identifying at least two similar images including a first image and a second image, identifying corresponding image regions in the first image and the second image, and assigning, using at least one processor, annotations to image regions in one or more images in the plurality of images by using a metric of fit indicative of a degree of match between the assigned annotations and the corresponding image regions. The metric of fit may depend on at least one annotation for each image in a subset of the plurality of images and the identified correspondence between image regions in the first image and the second image.

Claims

1. A method for semantically annotating images in a plurality of images, each image in the plurality of images comprising at least one image region, the method comprising:

identifying at least two similar images including a first image and a second image;

identifying corresponding image regions in the first image and the second image; and

iteratively assigning, using at least one processor, annotations to image regions in one or more images in the plurality of images by using a metric of fit indicative of a degree of match between the assigned annotations and the corresponding image regions,

the metric of fit depending on at least one annotation for each image in a subset of the plurality of images and the identified correspondence between image regions in the first image and the second image, the metric of fit computed using one or more spatial histograms of the at least one annotation for the each image.

2. The method of claim 1, wherein:

identifying corresponding image regions comprises identifying a first image region in the first image with a second image region in the second image; and

assigning the annotations to the image regions comprises iteratively assigning annotations to image regions based on the metric of fit,

wherein each iteration comprises assigning an annotation to the first image region based at least in part on an annotation assigned to the second image region.

3. The method of claim 2, wherein the first image comprises a third image region neighboring the first image region and wherein each iteration comprises:

assigning the annotation to the first image region further based on an annotation assigned to the third image region.

4. The method of claim 2, wherein each iteration comprises:

assigning the annotation to the first image region further based on the one or more spatial histograms of the annotation, wherein each of the one or more spatial histograms of the annotation is computed across multiple images in the plurality of images.

5. The method of claim 1, further comprising:

automatically obtaining the at least one annotation at least in part by obtaining at least one image-level annotation for an image in the subset of the plurality of images from text near the image in a document and/or webpage containing the image.

6. The method of claim 1, further comprising obtaining the at least one annotation by:

providing at least one image in the subset of the plurality of images to a user; and

obtaining, from the user, at least one image-region annotation for the at least one image.

7. The method of claim 1, wherein identifying the at least two similar images comprises:

computing a measure of similarity indicative of a degree of similarity between images for a pair of images;

identifying the pair of images as similar if the computed measure of similarity is greater than a predetermined threshold.

8. The method of claim 1, wherein identifying corresponding image regions in the first image and the second image comprises:

calculating a correspondence between image regions in the first image and image regions in the second image based on an objective function indicative of a degree of similarity between the image regions.

9. The method of claim 1, wherein the first image region is a pixel.

10. A system for enabling text-based search for images in a plurality of images based at least in part on image-level annotations associated with the images, the system comprising:

at least one processor configured to;

identify at least two similar images including a first image and a second image;

identify corresponding image regions in the first image and the second image; and

iteratively assign annotations to image regions in one or more images in the plurality of images by using a metric of fit indicative of a degree of match between the assigned annotations and the corresponding image regions,

the metric of fit depending on at least one annotation for each image in a subset of the plurality of images and the identified correspondence between image regions in the first image and the second image, the metric of fit computed using one or more spatial histograms of the at least one annotation for the each image.

11. The system of claim 10, wherein the at least one processor is configured to:

identify corresponding image regions by identifying a first image region in the first image with a second image region in the second image; and

assign the annotations to the image regions by iteratively assigning annotations to image regions based on the metric of fit,

wherein each iteration comprises assigning an annotation to the first image region based at least in part on an annotation assigned to the second image region.

12. The system of claim 11, wherein each iteration comprises calculating a statistical mapping between annotations and the image regions in the one or more images in the plurality of images.

13. The system of claim 12, wherein calculating the statistical mapping comprises estimating at least one parameter of a mixture model, wherein the mixture model comprises an element for each annotation in a set of annotations.

14. The system of claim 10, wherein the subset of the plurality of images comprises ten percent or less of the plurality of the images.

15. At least one tangible computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, perform a method for semantically annotating images in a plurality of images, each image in the plurality of images comprising one or more pixels, the method comprising:

obtaining at least one image-level annotation for each image in a subset of the plurality of images;

identifying at least two similar images including a first image and a second image;

identifying corresponding pixels in the first image and the second image; and

iteratively assigning annotations to pixels in one or more images in the plurality of images by using a metric of fit indicative of a degree of match between the assigned annotations and the corresponding pixels,

the metric of fit depending on at least one obtained image-level annotation and the identified correspondence between pixels in the first image and the second image, the metric of fit computed using one or more spatial histograms of the at least one annotation for the each image.

16. The at least one tangible computer-readable storage medium of claim 15, wherein

identifying corresponding pixels comprises identifying a first pixel in the first image with a second pixel in the second image;

and

assigning the annotations to the pixels comprises iteratively assigning annotations to the pixels based on the metric of fit,

wherein each iteration comprises assigning an annotation to the first pixel based at least in part on an annotation assigned to the second pixel.

17. The at least one tangible computer-readable storage medium of claim 16, wherein the first image comprises a third pixel near the first pixel and wherein each iteration comprises:

assigning the annotation to the first pixel further based on an annotation assigned to the third pixel.

18. The at least one tangible computer-readable storage medium of claim 16, wherein each iteration comprises calculating a statistical mapping between annotations and the image regions in the one or more images in the plurality of images.

19. The at least one tangible computer-readable storage medium of claim 15, further comprising:

automatically obtaining the at least one image-level annotation for an image in a webpage from text near the image on the webpage.

20. The at least one tangible computer-readable storage medium of claim 15, wherein obtaining the at least one image-level annotation includes:

providing at least one image in the subset of the plurality of images to a user; and

obtaining, from the user, at least one image-region annotation for the at least one image.