Image tag pair graph for image annotation

• US 9,146,941 B2
• Filed: 08/03/2012
• Issued: 09/29/2015
• Est. Priority Date: 08/03/2012
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Lin, Zhe, Brandt, Jonathan
• Primary Examiner(s): Park, Chan
• Assistant Examiner(s): RICE, ELISA M
• Application Number:
  US13/566,677
• Publication Number:
  US 20140037195A1
• Time in Patent Office: 1,152 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

An approach is described for automatically tagging a single image or multiple images. The approach, in one example embodiment, is based on a graph-based framework that exploits both visual similarity between images and tag correlation within individual images. The problem is formulated in the context of semi-supervised learning, where a graph modeled as a Gaussian Markov Random Field (MRF) is solved by minimizing an objective function (the image tag score function) using an iterative approach. The iterative approach, in one embodiment, comprises: (1) fixing tags and propagating image tag likelihood values from labeled images to unlabeled images, and (2) fixing images and propagating image tag likelihood based on tag correlation.

Claims

1. A computer-implemented method performed by one or more processors using a plurality of images, the method comprising:

creating a plurality of image tag pairs, each image and tag pair from the plurality of image tag pairs comprising an image from the plurality of images and a tag from a plurality of tags, each pair from the plurality of image tag pairs being assigned a probability value, the probability value indicating a likelihood of a tag from a pair being descriptive of an image from the pair;

for each subject tag from the plurality of tags;

examining subject tag pairs for similarity between respective images, the subject tag pairs being those pairs from the plurality of image tag pairs that include the subject tag, and

for each of the subject tag pairs, adjusting its assigned probability value, based on a result of the examining of the subject tag pairs;

for each image from the plurality of images;

examining subject image pairs for correlation between respective tags, the subject image pairs being those pairs from the plurality of image tag pairs that include the subject image, and

for each of the subject image pairs, adjusting its assigned probability value, based on a result of the examining of the subject image pairs; and

tagging unlabeled images from the plurality of images based on the assigned probability values adjusted in the examining of the subject tag pairs for similarity between respective images and the examining of the subject image pairs for correlation between respective tags.

2. The method of claim 1, comprising iterating the examining of the subject tag pairs, the examining of the subject image pairs, and the adjusting of probability values until respective probability values associated with the subject tag pairs and the subject image pairs become stable.

3. The method of claim 2, comprising:

the tagging the unlabeled images being responsive to determining that respective probability values associated with the subject tag pairs and the subject image pairs have become stable.

4. The method of claim 1, comprising assigning to a pair from the plurality of image tag pairs a first initial probability value if a tag from the pair is a user-selected tag for an image from the pair or a second initial probability value if the tag from the pair is not a user-selected tag for the image from the pair, wherein the first initial probability value indicates the highest probability of a likelihood of a tag from a pair being descriptive of an image from the pair.

5. The method of claim 1, wherein the plurality of tag image pairs, on which the operations of examining of the subject tag pairs, examining of the subject image pairs, and adjusting of probability values until respective probability values associated with the subject tag pairs and the subject image pairs become stable are performed, are represented by a single graph.

6. The method of claim 1, wherein images from the plurality images are user-supplied images.

7. The method of claim 1, wherein images from the plurality of images include user-supplied images and reference images.

8. The method of claim 1, wherein tags from the plurality of tags include user-supplied tags and reference tags.

9. The method of claim 1, wherein a tag from the plurality of tags include one or more classifiers, a classifier from the one or more classifiers indicating that the tag is related to another tag that includes the classifier.

10. The method of claim 9, wherein the examining of the subject image pairs comprises examining respective tags from the subject image pairs based on respective classifiers of the respective tags.

11. A computer-implemented system comprising:

an access module to access an image tag pair from a plurality of image tag pairs, each image tag pair from the plurality of image tag pairs comprising an image from a plurality of images and a tag from a plurality of tags, each pair from the plurality of image tag pairs being assigned a probability value, the probability value indicating a likelihood of a tag from a pair being descriptive of an image from the pair;

a fixed tag module to iterate through tags in the plurality of tags and, for each subject tag from the plurality of tags;

examine subject tag pairs for similarity between respective images, the subject tag pairs being those pairs from the plurality of image tag pairs that include the subject tag, and

for each of the subject tag pairs, adjust its assigned probability value, based on a result of the examining of the subject tag pairs;

a fixed image module to iterate through images in the plurality of images and, for each image from the plurality of images;

examine subject image pairs for correlation between respective tags, the subject image pairs being those pairs from the plurality of image tag pairs that include the subject image, and

for each of the subject image pairs, adjust its assigned probability value, based on a result of the examining of the subject image pairs; and

a tagging module to;

tag unlabeled images from the plurality of images based on the assigned probability values adjusted in the examining of the subject tag pairs for similarity between respective images and the examining of the subject image pairs for correlation between respective tags.

12. The system of claim 11, wherein the fixed tag module and the fixed image module are to iterate the examining of the subject tag pairs, the examining of the subject image pairs, and the adjusting of probability values until respective probability values associated with the subject tag pairs and the subject image pairs become stable.

13. The system of claim 12, wherein the tagging module tags the unlabeled images responsive to determining that respective probability values associated with the subject tag pairs and the subject image pairs have become stable.

14. The system of claim 11, wherein the access module is to assign to a pair from the plurality of image tag pairs a first initial probability value if a tag from the pair is a user-selected tag for an image from the pair or a second initial probability value if the tag from the pair is not a user-selected tag for the image from the pair, the first initial probability value indicates the highest probability of a likelihood of a tag from a pair being descriptive of an image from the pair.

15. The system of claim 11, wherein the plurality of tag image pairs, on which the operations of examining of the subject tag pairs, examining of the subject image pairs, and adjusting of probability values until respective probability values associated with the subject tag pairs and the subject image pairs become stable are performed, are represented by a single graph.

16. The system of claim 11, wherein images from the plurality images are user-supplied images.

17. The system of claim 11, wherein images from the plurality of images include user-supplied images and reference images.

18. The system of claim 11, wherein tags from the plurality of tags include user-supplied tags and reference tags.

19. The system of claim 11, wherein a tag from the plurality of tags include one or more classifiers, a classifier from the one or more classifiers indicating that the tag is related to another tag that includes the classifier.

20. A machine-readable non-transitory medium having instruction data to cause a machine to:

access an image tag pair from a plurality of image tag pairs, each image tag pair from the plurality of image tag pairs comprising an image from a plurality of images and a tag from a plurality of tags, each pair from the plurality of image tag pairs being assigned a probability value, the probability value indicating a likelihood of a tag from a pair being descriptive of an image from the pair;

iterate through tags in the plurality of tags and, for each subject tag from the plurality of tags;

examine subject tag pairs for similarity between respective images, the subject tag pairs being those pairs from the plurality of image tag pairs that include the subject tag, and

for each of the subject tag pairs, adjust its assigned probability value, based on a result of the examining of the subject tag pairs;

iterate through images in the plurality of images and, for each image from the plurality of images;

examine subject image pairs for correlation between respective tags, the subject image pairs being those pairs from the plurality of image tag pairs that include the subject image, and

for each of the subject image pairs, adjust its assigned probability value, based on a result of the examining of the subject image pairs; and

tag unlabeled images from the plurality of images based on the assigned probability values adjusted in the examining of the subject tag pairs for similarity between respective images and the examining of the subject image pairs for correlation between respective tags.