Method and apparatus for active annotation of multimedia content

• US 20040205482A1
• Filed: 01/24/2002
• Published: 10/14/2004
• Est. Priority Date: 01/24/2002
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Lin, Ching-Yung, Smith, John R., Naphade, Milind R., Tseng, Belle L., Basu, Sankar
• Application Number:
  US10/056,546
• Publication Number:
  US 20040205482A1
• Time in Patent Office: Days
• US Class Current: 715/500.1
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/7847 using low-level visual features of the video content
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

Semantic indexing and retrieval of multimedia content requires that the content is sufficiently annotated. However, the great volumes of multimedia data and diversity of labels make annotation a difficult and costly process. Disclosed is an annotation framework in which supervised training with partially labeled data is facilitated using active learning. The system trains a classifier with a small set of labeled data and subsequently updates the classifier by selecting a subset of the available data-set according to optimization criteria. The process results in propagation of labels to unlabeled data and greatly facilitates the user in annotating large amounts of multimedia content.

Claims

1. Method for generating persistent annotations of multimedia content, comprising one or more repetitions of the following steps:

actively selecting examples of multimedia content to be annotated by a user;

accepting input annotations from said user for said selected examples;

propagating said input annotations to other instances of multimedia content; and

storing said input annotations and said propagated annotations.

2. The method of claim 1, wherein the step of actively selecting is performed using a selection technique selected from the group consisting of:

deterministic and probabilistic.

3. The method of claim 2, wherein the step of actively selecting, which is performed deterministically or probabilistically, is based on explicit models and feature proximity/similarity measures, and returns one or more examples of multimedia content to be annotated.

4. The method of claim 2, wherein the step of actively selecting, which is performed deterministically or probabilistically, is based on implicit models and feature proximity/similarity measures, and returns one or more examples of multimedia content to be annotated.

5. The method of claim 1, wherein an optimization criterion for active selection includes one or more criteria selected from the group consisting of:

maximizing disambiguation, information measures, and confidence.

6. The method of claim 1, wherein the multimedia content comprises one or more types selected from the group consisting of:

images, audio, video, graphics, text, multimedia, Web pages, time series data, surveillance data, sensor data, relational data, and XML data.

7. The method of claim 1, wherein the input annotations are created by a user with reference to a vocabulary.

8. The method of claim 7, wherein the vocabulary contains one or more items selected from the group consisting of:

terms, concepts, labels, and annotations.

9. The method of claim 1, wherein the process of creating input annotations by the user involves multimodal interaction with the user using graphical, textual, and/or speech interface.

10. The method of claim 1, wherein the input annotations are created by means of steps selected from the group consisting of:

creating new annotations, deleting existing annotations, rejecting proposed annotations, and modifying annotations.

11. The method of claim 7, wherein the vocabulary is adaptively or dynamically organized and/or limited by the system or the user.

12. The method of claim 9, wherein the multimodal interaction involves speech recognition, gaze detection, finger pointing, expression detection, and/or effective computing methods for sensing a user'"'"'s state.

13. The method of claim 1, wherein the determination of the propagation of annotations is made deterministically or probabilistically and on the use of models for each annotation or for joint annotations.

14. The method of claim 2, wherein the models are created or learned automatically or semi-automatically and/or are updated adaptively from interaction with the user.

15. The method of claim 2, wherein the models are based on nearest neighbor voting or variants, parametric or statistical models, expert systems, rule-based systems, or hybrid techniques.

16. System for generating persistent annotations of multimedia content, comprising:

means for actively selecting examples of multimedia content to be annotated by a user;

means for accepting input annotations from said user for said selected examples;

means for propagating said input annotations to other instances of multimedia content; and

means for storing said input annotations and said propagated annotations.

17. The system of claim 16 wherein the means for actively selecting uses a selection technique selected from the group consisting of:

deterministic and probabilistic.

18. The system of claim 17, wherein the means for actively selecting, which uses a deterministic or probabilistic technique, is based on explicit models and feature proximity/similarity measures, and returns one or more examples of multimedia content to be annotated.

19. The system of claim 17, wherein the means for actively selecting, which uses a deterministic or probabilistic technique, is based on implicit models and feature proximity/similarity measures, and returns one or more examples of multimedia content to be annotated.

20. The system of claim 16, wherein an optimization criterion for active selection includes one or more criteria selected from the group consisting of:

maximizing disambiguation, information measures, and confidence.

21. The system of claim 16, wherein the multimedia content comprises one or more types selected from the group consisting of:

images, audio, video, graphics, text, multimedia, Web pages, time series data, surveillance data, sensor data, relational data, and XML data.

22. A computer program product in a computer readable medium for generating persistent annotations of multimedia content, the computer program product comprising instructions for performing one or more repetitions of the following steps:

actively selecting of examples of multimedia content to be annotated by a user;

accepting input annotations from said user for said selected examples;

propagating said input annotations to other instances of multimedia content; and

storing said input annotations and said propagated annotations.