Brief and high-interest video summary generation

• US 8,195,038 B2
• Filed: 10/24/2008
• Issued: 06/05/2012
• Est. Priority Date: 10/24/2008
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Liu, Zhu, Basso, Andrea, Gibbon, David Crawford, Shahraray, Behzad, Zavesky, Eric Martin
• Primary Examiner(s): Harvey, David
• Application Number:
  US12/290,033
• Publication Number:
  US 20100104261A1
• Time in Patent Office: 1,320 Days
• Field of Search: 386/278, 386/282, 386/343, 386/344, 386/350, 386/351, 715/719, 715/723, 715/730
• US Class Current: 386/344
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/322 used signal is digitally coded
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: H04N 5/147 Scene change detection

Abstract

A video is summarized by determining if a video contains one or more junk frames, modifying one or more boundaries of shots of the video based at least in part on the determination of if the video contains one or more junk frames, sampling a plurality of the shots of the video into a plurality of subshots, clustering the plurality of subshots with a multiple step k-means clustering, and creating a video summary based at least in part on the clustered plurality of subshots. The video is segmented into a plurality of shots and a keyframe from each of the plurality of shots is extracted. A video summary is created based on a determined importance of the subshots in a clustered plurality of subshots and a time budget. The created video summary is rendered by displaying playback rate information for the rendered video summary, displaying a currently playing subshot marker with the rendered video summary, and displaying an indication of similar content in the rendered video summary.

Claims

1. A method of video summarization comprising:

determining if a video contains a junk frame;

modifying a boundary of shots of the video based on the determining if a video contains a junk frame;

sampling a plurality of the shots of the video into a plurality of subshots;

clustering the plurality of subshots with a multiple step k-means clustering, comprising;

clustering the plurality of subshots;

determining a distance among the plurality of subshots as a weighted summation of a motion compensated matching error, a histogram difference in red, green, blue color space, and a histogram difference in hue, saturation, value color space;

determining a distance between a plurality of clusters as a maximum distance between pairs of subshots within the plurality of clusters;

merging the plurality of clusters if a determined distance between the plurality of clusters is smaller than a predetermined threshold; and

dividing a cluster in the plurality of clusters if a difference in a keyframe file size in the cluster exceeds a predetermined threshold; and

creating a video summary based on the clustering the plurality of subshots.

2. The method of claim 1 further comprising:

segmenting the video into a plurality of shots; and

extracting a keyframe from each of the plurality of shots.

3. The method of claim 1 wherein determining if a video contains a junk frame comprises:

extracting visual features from the video;

comparing the visual features to a predetermined threshold.

4. The method of claim 3 further comprising:

extracting acoustic features from an audio stream associated with the video; and

comparing the acoustic features to a predetermined threshold.

5. The method of claim 4 wherein the visual features are selected from a group comprising edge energies, color channel histogram variance, and color intensity and the acoustic features are selected from a group comprising volume, frequency centroid, frequency bandwidth, energy ratio in sub-bands, and voice activity.

6. The method of claim 1 wherein determining if a video contains a junk frame comprises:

detecting a clapper scene with a feature extractor;

extracting acoustic features from an audio stream associated with the video, wherein the acoustic features are selected from a group comprising volume, frequency centroid, frequency bandwidth, energy ratio in sub-bands, and zero crossing rate; and

comparing the acoustic features to a predetermined threshold.

7. The method of claim 1 wherein determining if a video contains a junk frame comprises:

detecting a clapper object in a frame of the video with a scale invariant feature transform feature set;

extracting acoustic features from an audio stream associated with the video, wherein the acoustic features are selected from a group comprising volume, frequency centroid, frequency bandwidth, energy ratio in sub-bands, and zero crossing rate; and

comparing the acoustic features to a predetermined threshold.

8. The method of claim 1 wherein creating a video summary based on the clustering the plurality of subshots comprises:

determining an importance of subshots in the plurality of subshots; and

creating a video summary based on the determining an importance of subshots in the plurality of subshots and a time budget.

9. A method of video summarization comprising:

determining if a video contains a junk frame;

modifying a boundary of shots of the video based on the determining if a video contains a junk frame;

sampling a plurality of the shots of the video into a plurality of subshots;

clustering the plurality of subshots with a multiple step k-means clustering;

creating a video summary based on the clustering the plurality of subshots;

rendering the video summary;

displaying playback rate information for a rendered video summary;

displaying a currently playing subshot marker with the rendered video summary; and

displaying an indication of similar content in the rendered video summary.

10. A non-transitory computer readable medium storing computer program instructions for video summarization, which, when executed on a processor, cause the processor to perform a method comprising:

determining if a video contains a junk frame;

modifying a boundary of shots of the video based on the determining if a video contains a junk frame;

sampling a plurality of the shots of the video into a plurality of subshots;

clustering the plurality of subshots with a multiple step k-means clustering, comprising;

clustering the plurality of subshots;

determining a distance among the plurality of subshots as a weighted summation of a motion compensated matching error, a histogram difference in red, green, blue color space, and a histogram difference in hue, saturation, value color space;

determining a distance between a plurality of clusters as a maximum distance between pairs of subshots within the plurality of clusters;

merging a plurality of clusters if a determined distance between the clusters is smaller than a predetermined threshold; and

dividing a cluster in the plurality of clusters if a difference in a keyframe file size in the cluster exceeds a predetermined threshold; and

creating a video summary based on the clustering the plurality of subshots.

11. The computer readable medium of claim 10 the method further comprising:

segmenting the video into a plurality of shots; and

extracting a keyframe from each of the plurality of shots.

12. The computer readable medium of claim 10 wherein the determining if a video contains a junk frame further comprises:

extracting visual features from the video;

comparing the visual features to a predetermined threshold.

13. The computer readable medium of claim 12, the method further comprising:

extracting acoustic features from an audio stream associated with the video; and

comparing the acoustic features to a predetermined threshold.

14. The computer readable medium of claim 13 wherein the visual features are selected from a group comprising edge energies, color channel histogram variance, and color intensity and the acoustic features are selected from a group comprising volume, frequency centroid, frequency bandwidth, energy ratio in sub-bands, and voice activity.

15. The computer readable medium of claim 10 wherein the determining if a video contains a junk frame further comprises:

detecting a clapper scene with a feature extractor;

extracting acoustic features from an audio stream associated with the video, wherein the acoustic features are selected from a group comprising volume, frequency centroid, frequency bandwidth, energy ratio in sub-bands, and zero crossing rate; and

comparing the acoustic features to a predetermined threshold.

16. The computer readable medium of claim 10 wherein the determining if a video contains a junk frame further comprises:

detecting a clapper object in a frame of the video with a scale invariant feature transform feature set;

extracting acoustic features from an audio stream associated with the video, wherein the acoustic features are selected from a group comprising volume, frequency centroid, frequency bandwidth, energy ratio in sub-bands, and zero crossing rate; and

comparing the acoustic features to a predetermined threshold.

17. The computer readable medium of claim 10 wherein the creating a video summary based on the clustering the plurality of subshots further comprises:

determining an importance of subshots in the plurality of subshots; and

creating a video summary based on the determining an importance of subshots in the plurality of subshots and a time budget.

18. The computer readable medium of claim 10, the method further comprising:

rendering the video summary;

displaying playback rate information for a rendered video summary;

displaying a currently playing subshot marker with the rendered video summary; and

displaying an indication of similar content in the rendered video summary.