Brief and high-interest video summary generation

• US 20100104261A1
• Filed: 10/24/2008
• Published: 04/29/2010
• Est. Priority Date: 10/24/2008
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Zavesky, Eric Martin, Gibbon, David Crawford, Shahraray, Behzad, Liu, Zhu, Basso, Andrea
• Granted Patent: US 8,195,038 B2
• Time in Patent Office: Days
• US Class Current: 386/241
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/322 used signal is digitally coded
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: H04N 5/147 Scene change detection

Abstract

A video is summarized by determining if a video contains one or more junk frames, modifying one or more boundaries of shots of the video based at least in part on the determination of if the video contains one or more junk frames, sampling a plurality of the shots of the video into a plurality of subshots, clustering the plurality of subshots with a multiple step k-means clustering, and creating a video summary based at least in part on the clustered plurality of subshots. The video is segmented into a plurality of shots and a keyframe from each of the plurality of shots is extracted. A video summary is created based on a determined importance of the subshots in a clustered plurality of subshots and a time budget. The created video summary is rendered by displaying playback rate information for the rendered video summary, displaying a currently playing subshot marker with the rendered video summary, and displaying an indication of similar content in the rendered video summary.

Claims

1. A method of video summarization comprising:

determining if a video contains one or more junk frames;

modifying one or more boundaries of shots of the video based at least in part on the determination of if the video contains one or more junk frames;

sampling a plurality of the shots of the video into a plurality of subshots;

clustering the plurality of subshots with a multiple step k-means clustering; and

creating a video summary based at least in part on the clustered plurality of subshots.

2. The method of claim 1 further comprising:

segmenting the video into a plurality of shots; and

extracting a keyframe from each of the plurality of shots.

3. The method of claim 1 wherein determining if a video contains one or more junk frames comprises:

extracting visual features from the video;

comparing the extracted visual features to a predetermined threshold.

4. The method of claim 3 further comprising:

extracting acoustic features from an audio stream associated with the video; and

comparing the extracted acoustic features to a predetermined threshold.

5. The method of claim 4 wherein the visual features are selected from the group of edge energies, color channel histogram variance, and color intensity and the acoustic features are selected from the group of volume, frequency centroid, frequency bandwidth, energy ratio in sub-bands, and voice activity.

6. The method of claim 1 wherein determining if a video contains one or more junk frames comprises:

detecting a clapper scene with a feature extractor;

extracting acoustic features from an audio stream associated with the video, wherein the acoustic features are selected from the group of volume, frequency centroid, frequency bandwidth, energy ratio in sub-bands, and zero crossing rate; and

comparing the extracted acoustic features to a predetermined threshold.

7. The method of claim 1 wherein determining if a video contains one or more junk frames comprises:

detecting a clapper object in one or more frames of the video with a scale invariant feature transform feature set;

extracting acoustic features from an audio stream associated with the video, wherein the acoustic features are selected from the group of volume, frequency centroid, frequency bandwidth, energy ratio in sub-bands, and zero crossing rate; and

comparing the extracted acoustic features to a predetermined threshold.

8. The method of claim 1 wherein clustering the plurality of subshots with a multiple step k-means clustering comprises:

clustering the plurality of subshots;

determining the distance among the plurality of subshots as a weighted summation of a motion compensated matching error, a histogram difference in red, green, blue color space, and a histogram difference in hue, saturation, value color space;

determining a distance between a plurality of clusters as a maximum distance between pairs of subshots within the plurality of clusters;

merging a plurality of clusters if the determined distance between the clusters is smaller than a predetermined threshold; and

dividing a cluster in the plurality of clusters if a difference in a,keyframe file size in the cluster exceeds a predetermined threshold.

9. The method of claim 1 wherein creating a video summary based at least in part on the clustered plurality of subshots comprises:

determining an importance of the subshots in the clustered plurality of subshots; and

creating a video summary based on the determined importance of the subshots in the clustered plurality of subshots and a time budget.

10. The method of claim 1 further comprising:

rendering the created video summary;

displaying playback rate information for the rendered video summary;

displaying a currently playing subshot marker with the rendered video summary; and

displaying an indication of similar content in the rendered video summary.

11. A computer readable medium comprising computer program instructions capable of being executed in a processor and defining the steps of:

determining if a video contains one or more junk frames;

modifying one or more boundaries of shots of the video based at least in part on the determination of if the video contains one or more junk frames;

sampling a plurality of the shots of the video into a plurality of subshots;

clustering the plurality of subshots with a multiple step k-means clustering; and

creating a video summary based at least in part on the clustered plurality of subshots.

12. The computer readable medium of claim 11 wherein the instructions further define the steps of:

segmenting the video into a plurality of shots; and

extracting a keyframe from each of the plurality of shots.

13. The computer readable medium of claim 11 wherein the instructions for determining if a video contains one or more junk frames further define the steps of:

extracting visual features from the video;

comparing the extracted visual features to a predetermined threshold.

14. The computer readable medium of claim 13 wherein the instructions further define the steps of:

extracting acoustic features from an audio stream associated with the video; and

comparing the extracted acoustic features to a predetermined threshold.

15. The computer readable medium of claim 14 wherein the visual features are selected from the group of edge energies, color channel histogram variance, and color intensity and the acoustic features are selected from the group of volume, frequency centroid, frequency bandwidth, energy ratio in sub-bands, and voice activity.

16. The computer readable medium of claim 11 wherein the instructions for determining if a video contains one or more junk frames further define the steps of:

detecting a clapper scene with a feature extractor;

extracting acoustic features from an audio stream associated with the video, wherein the acoustic features are selected from the group of volume, frequency centroid, frequency bandwidth, energy ratio in sub-bands, and zero crossing rate; and

comparing the extracted acoustic features to a predetermined threshold.

17. The computer readable medium of claim 11 wherein the instructions for determining if a video contains one or more junk frames further define the steps of:

detecting a clapper object in one or more frames of the video with a scale invariant feature transform feature set;

extracting acoustic features from an audio stream associated with the video, wherein the acoustic features are selected from the group of volume, frequency centroid, frequency bandwidth, energy ratio in sub-bands, and zero crossing rate; and

comparing the extracted acoustic features to a predetermined threshold.

18. The computer readable medium of claim 11 wherein the instructions for clustering the plurality of subshots with a multiple step k-means clustering further define the steps of:

clustering the plurality of subshots;

determining the distance among the plurality of subshots as a weighted summation of a motion compensated matching error, a histogram difference in red, green, blue color space, and a histogram difference in hue, saturation, value color space;

determining a distance between a plurality of clusters as a maximum distance between pairs of subshots within the plurality of clusters;

merging a plurality of clusters if the determined distance between the clusters is smaller than a predetermined threshold; and

dividing a cluster in the plurality of clusters if a difference in a keyframe file size in the cluster exceeds a predetermined threshold.

19. The computer readable medium of claim 11 wherein the instructions for creating a video summary based at least in part on the clustered plurality of subshots further define the steps of:

determining an importance of the subshots in the clustered plurality of subshots; and

creating a video summary based on the determined importance of the subshots in the clustered plurality of subshots and a time budget.

20. The computer readable medium of claim 11 wherein the instructions further define the steps of:

rendering the created video summary;

displaying playback rate information for the rendered video summary;

displaying a currently playing subshot marker with the rendered video summary; and

displaying an indication of similar content in the rendered video summary.