Method and system for video summarization

• US 9,355,635 B2
• Filed: 11/15/2011
• Issued: 05/31/2016
• Est. Priority Date: 11/15/2010
• Status: Active Grant

• Current Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Inventors: Gao, Jizhou, Huang, Yu, Yu, Hong Heather
• Primary Examiner(s): Saltarelli, Dominic D
• Application Number:
  US13/297,142
• Publication Number:
  US 20120123780A1
• Time in Patent Office: 1,659 Days
• Field of Search: 725/32
• US Class Current: 1/1
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: H04N 21/4358 for generating different versions, e.g. for different peripheral devices
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44016 involving splicing one content stream with another content stream, e.g. for substituting a video clip
• CPC Class Codes: H04N 21/44029 for generating different versions
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739

Abstract

A video summary method comprises dividing a video into a plurality of video shots, analyzing each frame in a video shot from the plurality of video shots, determining a saliency of each frame of the video shot, determining a key frame of the video shot based on the saliency of each frame of the video shot, extracting visual features from the key frame and performing shot clustering of the plurality of video shots to determine concept patterns based on the visual features. The method further comprises fusing different concept patterns using a saliency tuning method and generating a summary of the video based upon a global optimization method.

Claims

1. A method comprising:

dividing a video into a plurality of video shots;

analyzing each frame in a video shot from the plurality of video shots;

determining a saliency of each frame of the video shot;

determining a key frame of the video shot based on the saliency of each frame of the video shot;

extracting visual features from the key frame;

performing shot clustering of the plurality of video shots to determine concept patterns based on the visual features;

fusing different concept patterns using a saliency tuning method, wherein a saliency value obtained in the saliency tuning method is generated by fusing a visual saliency value, an audio saliency value, a motion saliency value and a spatial color saliency value; and

generating a summary of the video based upon a global optimization method.

2. The method of claim 1, further comprising generating the summary of the video based upon a dynamic programming method.

3. The method of claim 1, further comprising extracting audio features from the video shot, wherein extracting audio features comprises:

determining audio words from the video shot; and

performing clustering on the audio words.

4. The method of claim 3, wherein determining audio words comprises:

deriving a first set of audio words; and

calculating a saliency measure for each of the audio words.

5. The method of claim 4, wherein decomposing audio words into a series of overlapped short-time audio segments using matching pursuit decomposition method.

6. The method of claim 4, further comprising performing the saliency masking comprises comparing the saliency measure to a threshold and eliminating audio words having a saliency measure on one side of the threshold.

7. The method of claim 1, further comprising determining the saliency of each frame of the video shot using a dynamic programming method.

8. The method of claim 1, wherein determining the key frame comprises selecting a frame in the video shot with a balance between content highlights and information coverage.

9. The method of claim 1, wherein extracting visual features comprises determining a collection of visual words based on extracted visual features.

10. The method of claim 9, further comprising dividing the video into a plurality of shots by extracting visual, motion and audio features of each shot.

11. The method of claim 1, wherein performing shot clustering comprises

clustering to determine concept patterns among multiple shots;

and

grouping shots into concept categories.

12. A system comprising:

an extracting unit extracting a plurality of video shots from a video;

an analyzer determining a saliency of each frame of a video shot from the plurality of video shots;

a key frame determiner determining a key frame of the video shot based on the saliency of each frame in the video shot;

a visual feature extractor extracting visual features from the key frame;

a shot clustering block performing shot clustering of the plurality of video shots to determine concept patterns based on the visual features;

a saliency tuning block performing fusing different concept patterns using a saliency tuning method, wherein a saliency value obtained in the saliency tuning method is generated by fusing a visual saliency value, an audio saliency value, a motion saliency value and a spatial color saliency value; and

a summary generator generating a video summary based upon a global optimization method, wherein the global optimization method is carried out in a plurality of limited ranges, and wherein the plurality of limited ranges include a first range comprising a starting shot and a second range comprising an ending shot.

13. The system of claim 12, further comprising:

an audio feature determination block determining audio features from the video shot;

an audio saliency determination block determining saliency of the determined audio features;

an audio word clustering block clustering determined audio features; and

an audio and video concept aligning audio and video concept categories.

14. The system of claim 12, further comprising:

a video summary generator generating a video summary based on a dynamic programming method.

15. The system of claim 12, further comprising:

a video summary generator generating a video summary based upon a balance between maximum achievable saliency and video information coverage.

16. The system of claim 12, further comprising:

a video summary generator generating a video summary to preserve both content highlights and video information coverage.

17. A non-transitory computer readable medium with an executable program stored thereon, wherein the program instructs a microprocessor to perform the following steps:

analyzing each frame in a video shot from a plurality of video shots, analyzing comprising determining a saliency of each frame of the video shot;

determining a key frame of the video shot based on the saliency of each frame the video shot;

extracting visual features from the key frame;

performing shot clustering of the plurality of video shots to determine concept patterns based on the visual features;

fusing different concept patterns using a saliency tuning method, wherein a saliency value obtained in the saliency tuning method is generated by fusing a visual saliency value, an audio saliency value, a motion saliency value and a spatial color saliency value; and

generate a video summary based upon a global optimization method.

18. The non-transitory computer readable medium of claim 17, wherein the program instructs the microprocessor to further perform the steps of:

determining audio features of the video shot;

determining saliency of the audio features;

clustering determined audio features; and

aligning audio and video concept categories.

19. The non-transitory computer readable medium of claim 17, wherein the program instructs the microprocessor to further perform the step of generating a video summary based on the global optimization method.

20. The non-transitory computer readable medium of claim 17, wherein the program instructs the microprocessor to further perform the step of generating a video summary based on a dynamic programming method.