Blind summarization of video content

• US 7,143,352 B2
• Filed: 11/01/2002
• Issued: 11/28/2006
• Est. Priority Date: 11/01/2002
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electric Research Laboratories (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Divakaran, Ajay, Peker, Kadir A.
• Primary Examiner(s): SAX, STEVEN PAUL
• Application Number:
  US10/285,787
• Publication Number:
  US 20040085339A1
• Time in Patent Office: 1,488 Days
• Field of Search: 715719-720, 715721-726, 715771-773, 715/747, 715748-749, 715841-842, 715/968, 715/746, 715/811, 715853-855, 386/46, 386/52, 386/55, 386/68, 382/173, 382/181, 382/171, 382/232, 382/305, 382/224, 704/233, 345/719, 345/723
• US Class Current: 715/721
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

A method summarizes unknown content of a video. First, low-level features of the video are selected. The video is then partitioned into segments according to the low-level features. The segments are grouped into disjoint clusters where each cluster contains similar segments. The clusters are labeled according to the low-level features, and parameters characterizing the clusters are assigned. High-level patterns among the labels are found, and the these patterns are used to extract frames from the video according to form a content-adaptive summary of the unknown content of the video.

Claims

1. A method for summarizing unknown content of a video, comprising:

selecting low-level features of the video;

partitioning the video into segments according to the low-level features;

generating time-series data from the video based on the selected low-level features of the video;

self-correlating the time-series data to determine the similar segments

grouping the segments into a plurality of disjoint clusters, each cluster containing similar segments;

labeling the plurality of clusters with labels according to the low-level features;

finding high-level patterns among the labels; and

extracting frames from the video according to the high-level patterns to form a content-adaptive summary of the unknown content of the video.

2. The method of claim 1 wherein the summary is based on a combination of common and uncommon patterns.

3. The method of claim 1 wherein a production style of the content of the video is unknown.

4. The method of claim 1 wherein the video includes visual and audio data.

5. The method of claim 1 wherein the low-level features are selected from a group consisting of motion activity, color, texture, audio, semantic descriptors, and combinations thereof.

6. The method of claim 1 further comprising:

smoothing the time-series data.

7. The method of claim 1 further comprising:

weighting the time-series data.

8. The method of claim 1 further comprising:

associating the plurality of clusters by associations to infer the high-level patterns.

9. The method of claim 1 wherein all similar segments are found at a plurality of temporal resolutions.

10. The method of claim 1 further comprising:

self-correlating the time series data in a point cross-distance matrix to measure point cross-distances indicative of similarity of the segments.

11. The method of claim 10 wherein the point cross-distances are represented by substantially diagonal lines in the point cross-distance matrix.

12. The method of claim 1 further comprising:

time warping the time-series data to find the similar segments.

13. The method of claim 12 wherein the time warping is performed by dynamic programming.

14. The method of claim 1 further comprising:

correlating the labels in time.

15. The method of claim 14 wherein the time correlation is simultaneous.

16. The method of claim 1 further comprising:

generating conditions and results of occurrences of the labels to discover the high-level patterns.

17. The method of claim 8 wherein strong associations correspond to common and uncommon patterns.

18. The method of claim 1 wherein the grouping further comprises:

dividing overlapping segments so the overlapping segments have an identical overlap or no overlap.

19. The method of claim 1 further comprising:

assigning parameters to each cluster, the cluster parameters including a cluster compactness, a number of segments in the cluster, a total length of the segments in the cluster measured in frames, and an average distance between closest points of the segments to indicate a spreading of the segments in the video.

20. The method of claim 1 further comprising:

evaluating a cost function to determine a required length of the summary.