Feature identification of events in multimedia

• US 20050251532A1
• Filed: 08/20/2004
• Published: 11/10/2005
• Est. Priority Date: 05/07/2004
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Radhakrishnan, Regunathan, Divakaran, Ajay, Otsuka, Isao
• Granted Patent: US 7,302,451 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06F 16/7857 using texture G06F16/7837 takes precedence
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A method detects events in multimedia. Features are extracted from the multimedia. The features are sampled using a sliding window to obtain samples. A context model is constructed for each sample. The context models form a time series. An affinity matrix is determined from the time series models and a commutative distance metric between each pair of context models. A second generalized eigenvector is determined for the affinity matrix, and the samples are then clustered into events according to the second generalized eigenvector.

Claims

1. A method for detecting events in multimedia, comprising:

extracting features from the multimedia;

sampling the features using a sliding window to obtain a plurality of samples;

constructing a context model for each sample;

determining an affinity matrix from the models and a commutative distance metric between each possible pair of context models;

determining a second generalized eigenvector for the affinity matrix; and

clustering the plurality of samples into events according to the second generalized eigenvector.

2. The method of claim 1, in which the features represent usual events of a background process and unusual events of a foreground process.

3. The method of claim 1, further comprising:

labeling the features with labels.

4. The method of claim 3, in which the labels are discrete.

5. The method of claim 3, in which the labels are continuous.

6. The method of claim 1, in which the multimedia includes an audio portion and audio features are selected from the group consisting of pitch, amplitude, MeI frequency cepstral coefficients (MFCC), Modified Discrete Cosine Transform (MDCT) coefficients from an ac-3 stream, speech, and music.

7. The method of claim 1, in which the multimedia includes a video portion, and video features are selected multimedia from the group consisting of color, motion, texture, and MPEG-7 descriptors.

8. The method of claim 1, in which the sliding window has a length W_L, and a step size W_S, and wherein the length W_L determines a reliability of the context model, and the step size W_S determines a resolution of the events.

9. The method of claim 1, in which the second generalized eigenvector ranks the events.

10. The method of claim 9, further comprising:

summarizing the multimedia according to the ranked events.

11. The method of claim 2, in which the background process is memoryless, and the context model is an unconditional probability distribution function.

12. The method of claim 2, in which the background process has memory, and the context model is a hidden Markov model.

13. The method of claim 1, in which the context model is a Gaussian mixture model.

14. The method of claim 1, in which the sliding window has a length W_L, and the commutative distance metric d for each pair of context models λ

₁ and λ

₂ constructed from samples O₁ and O₂, respectively is

$\begin{array}{c}d<br><br>\left({\mathrm{\lambda <br><br>}}_1,{\mathrm{\lambda <br><br>}}_2\right)=<br><br>1/W_L(\log <br><br>\text{\hspace{1em}<br><br>}<br><br>P<br><br>\left(O_1❘<br><br>{\mathrm{\lambda <br><br>}}_1\right)+\log <br><br>\text{\hspace{1em}<br><br>}<br><br>P<br><br>\left(O_2❘<br><br>{\mathrm{\lambda <br><br>}}_2\right)-\\ <br><br>\log <br><br>\text{\hspace{1em}<br><br>}<br><br>P<br><br>\left(O_1❘<br><br>{\mathrm{\lambda <br><br>}}_2\right)-\log <br><br>\text{\hspace{1em}<br><br>}<br><br>P<br><br>\left(O_2❘<br><br>{\mathrm{\lambda <br><br>}}_1\right)).\end{array}$