Video skimming methods and systems

• US 9,171,578 B2
• Filed: 05/09/2011
• Issued: 10/27/2015
• Est. Priority Date: 08/06/2010
• Status: Expired due to Fees

• Current Assignee: VID SCALE Inc. (InterDigital, Inc.)
• Original Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Inventors: Lu, Taoran, Yuan, Zheng, Huang, Yu, Wu, Dapeng Oliver, Yu, Hong Heather
• Primary Examiner(s): Patel, Jayesh A
• Application Number:
  US13/103,810
• Publication Number:
  US 20120033949A1
• Time in Patent Office: 1,632 Days
• Field of Search: 382/305, 382/190, 382/177
• US Class Current: 1/1
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

In an embodiment, a method of creating a skimming preview of a video includes electronically receiving a plurality of video shots, analyzing each frame in a video shot from the plurality of video shots, where analyzing includes determining a saliency of each frame of the video shot. The method also includes determining a key frame of the video shot based on the saliency of each frame the video shot, extracting visual features from the key frame, performing shot clustering of the plurality of video shots to determine concept patterns based on the visual features, and generating a reconstruction reference tree based on the shot clustering. The reconstruction reference tree includes video shots categorized according to each concept pattern.

Claims

1. A method of creating a skimming preview of a video, the method comprising:

electronically receiving a plurality of video shots;

analyzing each frame in a video shot from the plurality of video shots, analyzing comprising determining a saliency of each frame of the video shot;

selecting a key frame of the video shot based on the saliency of each frame of the video shot after determining the saliency of each frame in the video shot;

extracting visual features from the key frame after selecting the key frame;

performing shot clustering of the plurality of video shots to determine concept patterns based on the visual features; and

generating a reconstruction reference tree based on the shot clustering, the reconstruction reference tree comprising video shots categorized according to each concept pattern, wherein generating the reconstruction reference tree comprises categorizing video shots within concept categories ordered according to concept importance, and ordering video shots within each concept category according to saliency.

2. The method of claim 1, further comprising generating a video skimming preview based on the reconstruction reference tree.

3. The method of claim 1, further comprising extracting audio features from the video shot.

4. The method of claim 3, wherein extracting the audio features comprises:

determining audio words from the video shot; and

performing clustering on the audio words.

5. The method of claim 4, wherein determining audio words comprises:

deriving a first set of audio words;

calculating a saliency measure for each of the audio words;

performing a saliency masking of the audio words, wherein performing the saliency masking comprises comparing the saliency measure to a threshold and eliminating audio words having a saliency measure on one side of the threshold; and

decomposing audio words into a series of predefined waveform bases.

6. The method of claim 5, wherein decomposing audio words into a series of predefined waveform bases comprises using a matching pursuit (MP) based histogram.

7. The method of claim 5, wherein calculating the saliency measure comprises performing a spectral centroid analysis of low level audio features.

8. The method of claim 1, further comprising:

determining shot boundaries of the plurality of video shots; and

segmenting the plurality of video shots based on the determined shot boundaries.

9. The method of claim 1, wherein determining the saliency of each frame of the video shot comprises using information from motion and color channels from the video shot.

10. The method according to claim 1, wherein selecting the key frame comprises selecting a frame in the video shot having a highest saliency.

11. The method according to claim 1, wherein extracting visual features from the key frame comprises:

defining an active region on the key frame by thresholding the determined saliency; and

performing a feature detection on the active region.

12. The method according to claim 1, wherein extracting visual features comprises determining a collection of visual words based on the extracted visual features.

13. The method according to claim 12, wherein determining the collection of visual words further comprises deriving a histogram of the visual words to represent the video shot.

14. The method according to claim 1, wherein performing shot clustering comprises

clustering to determine concept patterns among multiple shots;

and

grouping shots into concept categories.

15. The method according to claim 14, wherein performing shot clustering further comprises aligning audio and video concept categories.

16. The method according to claim 15, further comprising ordering concept categories in order of importance, wherein importance is determined by a number of shots in each concept category and by saliency of each shot in each concept category.

27. The method of claim 1, further comprising determining the concept importance comprising determining a total number of frames in a shot having a same concept.

28. The method of claim 1, further comprising:

selecting shots starting from categories of highest importance and shots of highest saliency within the categories of highest importance; and

generating the skimming preview based on the selected shots.

17. A method of generating a skimming preview of a video, the method comprising:

electronically receiving a reconstruction reference tree comprising video shots categorized within concept categories ordered according to concept importance, wherein video shots within each concept category is ordered according to saliency;

determining a skimming ratio;

selecting shots starting from categories of highest importance and shots of highest saliency within the categories of highest importance; and

generating the skimming preview based on the selected shots.

18. The method of claim 17, further comprising:

selecting frames within the selected shots having a highest saliency; and

using the selected frames within the selected shots having the highest saliency to generate the skimming preview.

19. The method of claim 18, wherein selecting frames within the selected shots having a highest saliency comprises comparing the saliency of the selected shots to a threshold.

20. A system for creating a skimming preview of a video, the system comprising:

an I/O port electronically receiving a plurality of video shots; and

a hardware circuit coupled to the I/O port, the hardware circuit comprising

a saliency analyzer determining a saliency of each frame of a video shot from the plurality of video shots,

a key frame selector selecting a key frame of the video shot based on the saliency of each frame in the video shot after the saliency analyzer determines the saliency of each frame in the video shot,

a visual feature extractor extracting visual features from the key frame after the key frame selector selects the key frame,

a shot clustering block performing shot clustering of the plurality of video shots to determine concept patterns based on the visual features, and

a reconstruction reference tree generator generating a reconstruction reference tree based on the shot clustering, the reconstruction reference tree comprising video shots categorized according to each concept pattern, wherein the reconstruction reference tree generator is configured to generate the reconstruction reference tree by categorizing video shots within concept categories ordered according to concept importance, and ordering video shots within each concept category according to saliency.

21. The system of claim 20, further comprising an audio feature determination block determining audio features from the video shot;

an audio saliency determination block determining saliency of the determined audio features;

an audio word clustering block clustering determined audio features; and

an audio and video concept aligner aligning audio and video concept categories.

22. The system of claim 20, further comprising a video skimming preview generator generating a skimming preview based on the reconstruction reference tree.

26. The system of claim 20, wherein the hardware circuit comprises a processor configured to implement the saliency analyzer, key frame selector, visual feature extractor, shot clustering block and reconstruction reference tree generator.

29. The system of claim 20, wherein the hardware circuit is further configured to determine the concept importance by determining a total number of frames in a shot having a same concept.

30. The system of claim 20, wherein the hardware circuit is further configured to:

select shots starting from categories of highest importance and shots of highest saliency within the categories of highest importance; and

generate the skimming preview based on the selected shots.

23. A non-transitory computer readable medium with an executable program stored thereon, wherein the program instructs a microprocessor to perform the following steps:

analyzing each frame in a video shot from a plurality of video shots, analyzing comprising determining a saliency of each frame of the video shot;

selecting a key frame of the video shot based on the saliency of each frame of the video shot after determining the saliency of each frame in the video shot;

extracting visual features from the key frame after selecting the key frame;

performing shot clustering of the plurality of video shots to determine concept patterns based on the visual features; and

generating a reconstruction reference tree based on the shot clustering, the reconstruction reference tree comprising video shots categorized according to each concept pattern, wherein generating the reconstruction reference tree comprises categorizing video shots within concept categories ordered according to concept importance, and ordering video shots within each concept category according to the saliency.

24. The non-transitory computer readable medium of claim 23, wherein the program instructs the microprocessor to further perform the steps of:

determining audio features of the video shot;

determining a saliency of the determined audio features;

clustering determined audio features; and

aligning audio and video concept categories.

25. The non-transitory computer readable medium of claim 23, wherein the program instructs the microprocessor to further perform the step of generating a skimming preview based on the reconstruction reference tree.

31. The non-transitory computer readable medium of claim 23, wherein the program instructs the microprocessor to further perform the step of determining the concept importance by determining a total number of frames in a shot having a same concept.

32. The non-transitory computer readable medium of claim 23, wherein the program instructs the microprocessor to further perform the steps of:

selecting shots starting from categories of highest importance and shots of highest saliency within the categories of highest importance; and

generating a skimming preview based on the selected shots.