TAGGING OVER TIME: REAL-WORLD IMAGE ANNOTATION BY LIGHTWEIGHT METALEARNING

• US 20090083332A1
• Filed: 09/19/2008
• Published: 03/26/2009
• Est. Priority Date: 09/21/2007
• Status: Abandoned Application

• Current Assignee: Penn State Research Foundation (Pennsylvania State University)
• Original Assignee: Penn State Research Foundation (Pennsylvania State University)
• Inventors: Li, Jia, Wang, James Z., Datta, Ritendra, Joshi, Dhiraj
• Application Number:
  US12/234,159
• Publication Number:
  US 20090083332A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 18/2178 based on feedback of a supervisor

Abstract

A principled, probabilistic approach to meta-learning acts as a go-between for a ‘black-box’ image annotation system and its users. Inspired by inductive transfer, the approach harnesses available information, including the black-box model'"'"'s performance, the image representations, and a semantic lexicon ontology. Being computationally ‘lightweight.’ the meta-learner efficiently re-trains over time, to improve and/or adapt to changes. The black-box annotation model is not required to be re-trained, allowing computationally intensive algorithms to be used. Both batch and online annotation settings are accommodated. A “tagging over time” approach produces progressively better annotation, significantly outperforming the black-box as well as the static form of the meta-learner, on real-world data.

Claims

1. A method of annotating an image, comprising the steps of:

receiving one or more annotations of an image from an existing, black box image annotation system;

providing additional annotations of the image using the annotations provided by the black box system and other available resources;

computing the probability that each additional annotation is an accurate annotation for the image; and

annotating the image using those annotations having the highest probability.

2. The method of claim 1, wherein the existing, black box image annotation system is a batch annotation system.

3. The method of claim 1, wherein the existing, black box image annotation system is an online annotation system.

4. The method of claim 1, wherein the available resources includes ground-truth annotations or tags.

5. The method of claim 1, wherein the available resources includes a semantic lexicon.

6. The method of claim 1, wherein the available resources includes the visual content of the image.

7. The method of claim 1, wherein the available resources includes the performance of the black-box system.

8. The method of claim 1, wherein the step of computing the probability that an additional annotation is accurate includes computing the probability that the annotation is an actual ground-truth tag.

9. The method of claim 1, wherein using those annotations having the highest probability includes using the top-ranked annotations.

10. The method of claim 1, wherein using those annotations having the highest probability includes thresholding the top percentile of the annotations.

11. The method of claim 1, wherein the step of providing additional annotations of the image includes guessing.

12. The method of claim 1, wherein the step of computing the probability that each additional annotation is an accurate annotation for the image includes making independent decisions with respect to each word comprising an annotation.

13. The method of claim 1, wherein the black box annotation system is an online system of the type wherein images and user tags enter the system as a temporal sequence.

14. The method of claim 1, wherein the step of providing additional annotations includes the step of providing initial training annotations.

15. The method of claim 14, wherein:

the step of providing additional annotations includes the step of providing initial training annotations; and

including the step of smoothing the computed probabilities to account for sparsity associated with available annotations.

16. The method of claim 15, wherein the step of smoothing is an interpolation-based.

17. The method of claim 15, wherein the step of smoothing is based upon similarity-based smoothing to model word pair co-occurrences.

18. The method of claim 1, further including the step of re-training following the annotation of a plurality of images.

19. The method of claim 1, wherein the re-training is based upon a persistent memory model.

20. The method of claim 1, wherein the re-training is based upon a transient memory model.