COLLECTIVE MEDIA ANNOTATION USING UNDIRECTED RANDOM FIELD MODELS

• US 20080112625A1
• Filed: 11/10/2006
• Published: 05/15/2008
• Est. Priority Date: 11/10/2006
• Status: Active Grant

• Current Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Cooper, Matthew L.
• Granted Patent: US 7,986,842 B2
• Time in Patent Office: Days
• US Class Current: 382/228
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models

Abstract

In an embodiment, the present invention relates to a method for semantic analysis of digital multimedia. In an embodiment of the invention, low level features are extracted representative of one or more concepts. A discriminative classifier is trained using these low level features. A collective annotation model is built based on the discriminative classifiers. In various embodiments of the invention, the frame work is totally generic and can be applied with any number of low-level features or discriminative classifiers. Further, the analysis makes no domain specific assumptions, and can be applied to activity analysis or other scenarios without modification. The framework admits the inclusion of a broad class of potential functions, hence enabling multi-modal analysis and the fusion of heterogeneous information sources.

Claims

1. A method for detecting one or more concept in multimedia comprising:

(a) extracting low level features representative of the one or more concept;

(b) training a discriminative classifier for each concept using a set of the low level features;

(c) building a collective annotation model combining each of the discriminative classifiers;

(d) defining one or more interaction potential to model interdependence between related concepts; and

(e) detecting the presence/absence of one or more concepts based on the collective annotation model and the defined interaction potentials.

2. The method of claim 1, wherein the multimedia includes one or more digital video frames.

3. The method of claim 2, wherein the multimedia further includes one or more forms of data selected from the group consisting of aligned digital text, tag information, text transcripts and web page links.

4. The method of claim 1, wherein the set of low level features are selected from the group consisting of color histograms, texture features, edge features, motion analysis, face detection and aligned text data.

5. The method of claim 1, wherein the discriminative classifier is a support vector machine.

6. The method of claim 5, wherein the output of the support vector machine is transformed to a probability using a logistic mapping.

7. The method of claim 1, wherein the collective annotation model is selected from the group consisting of discriminative random field (DRF) model, conditional random field (CMU) model, discriminative output independent concept detection (SVM) model, inter concept co-occurrence (CML+1) model and concept feature co-occurrence (CMLT+1) model.

8. The method of claim 1, wherein the defined interaction potential is a function of each pair of concepts Yi.Yj.

9. The method of claim 8, wherein the defined interaction potential distinguishes all four binary combinations of Yi and Yj.

10. The method of claim 1, wherein interaction potential is a function of each pair of concepts Yi.Yj and low level feature data.

11. The method of claim 1, wherein a discriminative classifier applies to a single concept, wherein a set of discriminative classifiers applies to a set of concepts.

12. The method of claim 11, wherein the set of discriminative classifiers is integrated in a framework for collective multimedia annotation.

13. The method of claim 1, further comprising supplying one or both of a confidence measure and a ranking associated with the identified concepts.

14. The method of claim 13, wherein one or both of the confidence measure and the ranking can vary with time.

15. The method of claim 13, wherein one or both of the confidence measure and the ranking can be used for recommending an annotation to a user in the form of a ranked list.

16. The method claim 1, further comprising leveraging pre-trained concept detectors to improve detection of concepts.

17. The method of claim 1, further comprising employing approximate inference strategies to improve detection of concepts.

18. The method of claim 1, further comprising quantization of the low level features during the training.

19. A system to identifying one or more concepts in digital media comprising:

a processing component for extracting low level features representative of one or more concepts;

a processing component for training a discriminative classifier for each concept using a set of the low level features;

a processing component capable of building a collective annotation model based on each of the discriminative classifiers;

one or more defined interaction potential to identify related concepts; and

a processing component capable of identifying one or more concepts based on the collective annotation model and the defined interaction potentials.

20. A machine readable medium having instructions stored thereon that when executed by a processor cause a system to:

extract low level features representative of the one or more concept;

train a discriminative classifier for each concept using a set of the low level features;

build a collective annotation model based on each of the discriminative classifiers;

define one or more interaction potential to identify related concepts, and

identify one or more concepts based on the collective annotation model and the defined interaction potentials.