Usual event detection in a video using object and frame features

• US 7,426,301 B2
• Filed: 06/28/2004
• Issued: 09/16/2008
• Est. Priority Date: 06/28/2004
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Porikli, Fatih M.
• Primary Examiner(s): Bella; Matthew C.
• Assistant Examiner(s): Strege; John B
• Application Number:
  US10/878,958
• Publication Number:
  US 20050286774A1
• Time in Patent Office: 1,541 Days
• Field of Search: 382/181, 382/190, 382/224, 382/225, 348/143
• US Class Current: 382/181
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

The invention provides a method for detecting usual events in a video. The events are detected by first constructing an aggregate affinity matrix from features of associated items extracted from the video. The affinity matrix is decomposed into eigenvectors, and the eigenvectors are used to reconstruct approximate estimates of the aggregate affinity matrix. Each matrix is clustered and scored, and the clustering that yields the highest scores is used to detect usual events.

Claims

1. A method for detecting usual events in a video, the video including a plurality of items, comprising:

extracting a set of features for each item in the video, in which each item is an object, and the features are object-based, and in which the features are associated a trajectory of the object;

constructing an affinity matrix for each feature according to the items;

aggregating the affinity matrices into an aggregate affinity matrix;

decomposing the aggregate affinity matrix into an set of eigenvectors, in a first to last order;

reconstructing a plurality of approximate aggregate affinity matrices, wherein a first approximate aggregate affinity matrix is reconstructed from the first eigenvector, and each next approximate aggregate affinity matrix includes one additional one of the eigenvectors in the first to last order, and a last approximate aggregate affinity matrix is reconstructed from all of the eigenvectors;

clustering items associated with each approximate aggregate affinity matrix into clusters;

evaluating each approximate aggregate affinity matrix to determine a validity score for each approximate aggregate affinity matrix; and

selecting the approximate aggregate affinity matrix with a highest validity score as the clustering of the items associated with usual events;

emitting the trajectories as the observable output of the usually events associated with the objects.

2. The method of claim 1, in which the features are visual features.

3. The method of claim 1, in which the features are audio features.

4. The method of claim 1, in which the items are frames, and the features are frame-based.

5. The method of claim 1, in which the features include histograms, hidden Markov models, and scalar values of low-level features.

6. The method of claim 1, in which features are features are weighted and normalized during the construction.

7. The method of claim 1, in which the reconstructing is a cross-product of the eigenvectors.

8. The method of claim 1, in which the trajectories are of unequal duration.