Unsupervised learning of video structures in videos using hierarchical statistical models to detect events

• US 20050131869A1
• Filed: 12/12/2003
• Published: 06/16/2005
• Est. Priority Date: 12/12/2003
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Chang, Shih-Fu, Divakaran, Ajay, Xie, Lexing
• Granted Patent: US 7,313,269 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06F 16/7864 using domain-transform features, e.g. DCT or wavelet transform coefficients
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

A method learns a structure of a video, in an unsupervised setting, to detect events in the video consistent with the structure. Sets of features are selected from the video. Based on the selected features, a hierarchical statistical model is updated, and an information gain of the hierarchical statistical model is evaluated. Redundant features are then filtered, and the hierarchical statistical model is updated, based on the filtered features. A Bayesian information criteria is applied to each model and feature set pair, which can then be rank ordered according to the criteria to detect the events in the video.

Claims

1. A method for learning a structure of a video to detect events in the video consistent with the structure, comprising:

selecting sets of features from the video;

updating a hierarchical statistical model for each set of features;

evaluating an information gain of the hierarchical statistical model;

filtering redundant features;

updating the hierarchical statistical model based on the filtered features;

applying a Bayesian information criteria to each model and feature set pair; and

rank ordering the model and feature set pairs to learn the structure and detect the events in the video in an unsupervised manner.

2. The method of claim 1, in which the hierarchical statistical model uses Gaussian mixtures.

3. The method of claim 1, in which the hierarchical statistical model uses hidden Markov models.

4. The method of claim 3, in which states of events in the video are modeled as low-level hidden Markov models in the hierarchical hidden Markov model, and the events are modeled as a high-level Markov chain in the hierarchical hidden Markov model.

5. The method of claim 1, in which the features include dominant color ratios, motion intensity, a least-square estimates of camera translation, audio volume, spectral roll-off, low-band energy, high-band energy, zero-crossing rate (ZCR).

6. The method of claim 1, in which the features are filtered with a Markov blanket.

7. The method of claim 1, in which the evaluating is performed using expectation maximization and a Markov chain Monte Carlo method.