Method for detecting scene boundaries in genre independent videos

• US 20080193017A1
• Filed: 02/14/2007
• Published: 08/14/2008
• Est. Priority Date: 02/14/2007
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Divakaran, Ajay, Wilson, Kevin W., Goela, Naveen, Niu, Feng, Otsuka, Isao
• Granted Patent: US 7,756,338 B2
• Time in Patent Office: Days
• US Class Current: 382/190
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

A computer implemented method detects scene boundaries in videos by first extracting feature vectors from videos of different genres. The feature vectors are then classified as scene boundaries using a support vector machine. The support vector machine is trained to be independent of the different genres of the videos.

Claims

1. A computer implemented method for detecting scene boundaries in videos, comprising the step of:

extracting feature vectors from videos of different genres; and

classifying the feature vectors as scene boundaries using a support vector machine, and in which the support vector machine is trained to be independent of the different genre of the videos.

2. The method of claim 1, further comprising:

extracting audio features from an audio signal of the videos;

extracting visual features from frames of the videos; and

combining the audio and visual features into the features vectors.

3. The method of claim 1, in which the videos are compressed.

4. The method of claim 1, further comprising:

feeding back results of the classifying to improve the feature extracting.

5. The method of claim 1, further comprising:

segmenting the videos according to the scene boundaries.

6. The method of claim 1, further comprising:

labeling the scene boundaries in training videos to train the support vector machine.

7. The method of claim 2, further comprising:

extracting Mel-frequency cepstral coefficients (MFCCs) from the audio signals; and

classifying the audio signals into semantic classes.

8. The method of claim 7, in which the semantic classes include music, speech, laughter, and silence.

9. The method of claim 8, in which the speech includes male and female speech.

10. The method of claim 2, in which the visual features include shot boundaries.

11. The method of claim 7, in which each feature vector includes variable x₁, x₂, x₃ indicating a number of the audio classes labels within a time window of duration [t−

W_L, t], where W_L is about fourteen seconds, and variables x₄, x₅, x₆ indicating a number of the audio classes in a window of duration

$\left[t-\frac{W_L}{2},t+\frac{W_L}{2}\right],$

and variables x₇, x₈, x₉ indicating a number of audio classes within a window [t, t+W_L], and variables x₁₀, x₁₁ are a Bhattacharyya shape and a Mahalanobis distance between the MFCC coefficients for the window [t−

W_L, t] and window [t, t+W_L], respectively, and variable x₁₂ is twice an average number of shot boundaries present in the video within a window [t−

W_L, t+W_L].

12. The method of claim 11, in which the Bhattacharyya shape is

$D_{\mathrm{shape}}=\frac{1}{2}$

ln 



C i + C j 2 



C i 

1 2 



C j 

1 2 ,

and

the Mahalanobis distance is

$D_{\mathrm{mahal}}=\frac{1}{8}<br><br>{\left({\mathrm{\mu <br><br>}}_i-{\mathrm{\mu <br><br>}}_j\right)}^T<br><br>{\left(\frac{C_i+C_j}{2}\right)}^{-1}<br><br>\left({\mathrm{\mu <br><br>}}_i-{\mathrm{\mu <br><br>}}_j\right),$

where covariance matrices C_i and C_j, and means μ

_i and μ

_j represent diagonal covariances and means of the MFCC vectors before and after time t.

13. The method of claim 1, further comprising:

transforming the feature vectors to a higher dimensional feature space using a kernel function.

14. The method of claim 13, in which the kernel function is a radial basis kernel.