Video mining using unsupervised clustering of video content

• US 20040085323A1
• Filed: 11/01/2002
• Published: 05/06/2004
• Est. Priority Date: 11/01/2002
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Divakaran, Ajay, Peker, Kadir A.
• Granted Patent: US 7,375,731 B2
• Time in Patent Office: Days
• US Class Current: 345/581
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06F 16/7857 using texture G06F16/7837 takes precedence
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content

Abstract

A method mines unknown content of a video by first selecting one or more low-level features of the video. For each selected feature, or combination of features, time series data is generated. The time series data is then self-correlated to identify similar segments of the video according to the low-level features. The similar segments are grouped into clusters to discover high-level patterns in the unknown content of video.

Claims

1. A method for mining content of a video, comprising:

selecting a low-level feature of the video;

generating time-series data from the content of the video based on the selected low-level feature;

self-correlating the time-series data to determine similar segments of the video according to the low-level feature; and

clustering the similar segments to discover high-level patterns in the content of the video.

2. The method of claim 1 wherein the content of the video is unknown.

3. The method of claim 1 wherein a production style of the video is unknown.

4. The method of claim 1 wherein the video includes video and audio data.

5. The method of claim 1 wherein the low-level feature is selected from a group consisting of motion activity, color, texture, audio, semantic descriptors, and combinations thereof.

6. The method of claim 1 further comprising:

thresholding the low-level feature.

7. The method of claim 1 further comprising:

smoothing the time-series data.

8. The method of claim 1 further comprising:

weighting the time-series data.

9. The method of claim 1 further comprising:

selecting a plurality of low-level features; and

generating and self-correlating for each of the selected low-level features to generate a plurality of sets of clusters.

10. The method of claim 9 further comprising:

associating the plurality of sets of clusters to infer the high-level patterns.

11. The method of claim 1 wherein all similar segments are found at a plurality of temporal resolutions.

12. The method of claim 11 further comprising:

implicitly constructing multi-resolution windows for the time-series data.

13. The method of claim 12 wherein a distance between two windows of size w is measured as

14. The method of claim 1 further comprising:

self-correlating the time-series data in a point cross-distance matrix to measure the point cross-distances.

15. The method of claim 14 wherein the point cross-distances are represented by substantially diagonal lines in the point cross-distance matrix.

16. The method of claim 1 further comprising:

time warping the time-series data to find the similar segments.

17. The method of claim 16 wherein the time warping is performed by dynamic programming.

18. The method of claim 1 further comprising:

merging the similar segments using a Hough transform on the time-series data.

19. The method of claim 1 further comprising:

labeling the clusters.

20. The method of claim 19 further comprising:

correlating the labels in time.

21. The method of claim 20 wherein the time correlation is simultaneous.

22. The method of claim 20 wherein the time correlation is time shifted.

23. The method of claim 20 further comprising:

generating conditions and results of occurrences of the labels to discover the high-level patterns.

24. The method of claim 1 further comprising:

summarizing the video according to the high-level patterns.

25. The method of claim 1 wherein the self-correlating further comprises:

measuring point cross-distances in the time series-data.

26. The method of claim 1 wherein the self-correlating further comprises:

forming a cross-product of the time-series data.

27. The method of claim 1 wherein the time-series data is a color histogram of consecutive frames in the video, and the self-correlating further comprises:

intersecting the color histogram with itself.

28. The method of claim 1 wherein the time-series data is a function of the low-level feature and frames of the video.

29. The method of claim 1 wherein the time-series data is multi-dimensional.