Systems and methods for generating a video summary

• US 20040088723A1
• Filed: 11/01/2002
• Published: 05/06/2004
• Est. Priority Date: 11/01/2002
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Li, Mingjing, Zhang, Hong-Jiang, Lu, Lie, Ma, Yu-Fei
• Application Number:
  US10/286,527
• Publication Number:
  US 20040088723A1
• Time in Patent Office: Days
• US Class Current: 725/19
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: H04N 21/25891 being end-user preferences retrieval of video data in a video database based on user preferences G06F16/739; arrangements for recognizing users' preferences H04H60/46; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/8453 by locking or enabling a set of features, e.g. optional functionalities in an executable program
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739

Abstract

Systems and methods to generate a video summary of a video data sequence are described. In one aspect, key-frames of the video data sequence are identified independent of shot boundary detection. A static summary of shots in the video data sequence is then generated based on key-frame importance. For each shot in the static summary of shots, dynamic video skims are calculated. The video summary consists of the calculated dynamic video skims.

Claims

1. A method for generating a video summary of a video data sequence, the method comprising:

identifying, independent of shot boundary detection, key-frames of the video data sequence;

generating, based on determined key-frame importance, a static summary of shots from the video data sequence; and

calculating, for each shot in the static summary of shots, one or more dynamic video skims for the shot, an aggregate of calculated dynamic video skims being the video summary.

2. A method as recited in claim 1, wherein identifying the key frames further comprises analyzing attention values provided by a comprehensive user attention model to identify the key frames, the comprehensive user attention model being based at least on integrated visual and audio attention models.

3. A method as recited in claim 1, wherein generating the static summary of shots further comprises:

responsive to determining that there are more key-frames than a threshold number of allowed shots;

assigning each shot an importance value based on maximum key-frame importance in the shot, a shot without a key-frame being assigned a lowest importance value;

dropping shots with importance values that are low as compared to respective importance values of other shots; and

selecting only non-dropped shots for the static summary of shots.

4. A method as recited in claim 1, wherein a shot includes one or more key-frames, and wherein calculating further comprises:

determining a skim ratio for the shot; and

for each of the one or more key-frames in the shot, selecting a skim segment around the key-frame according to the skim ratio.

5. A method as recited in claim 1, wherein calculating further comprises adjusting dynamic skim segment boundaries according to one or more sentence boundaries.

6. A method as recited in claim 1, wherein calculating further comprises, given a skim ratio for the shot, creating each of the one or more dynamic video skims with a respective length that is based on length of the shot and number of key-frames in the shot.

10. A method as recited in claim 1, wherein identifying the key frames further comprises analyzing a comprehensive user attention model to identify the key frames, the comprehensive user attention model being based at least on integrated visual and audio attention models.

7. A computer-readable memory comprising computer-program instructions executable by a processor to generate a video summary of a video data sequence, the computer-program instructions comprising instructions for:

identifying, independent of shot boundary detection, key-frames of the video data sequence;

generating, based on determined key-frame importance, a static summary of shots from the video data sequence; and

calculating, for each shot in the static summary of shots, one or more dynamic video skims for the shot, an aggregate of calculated dynamic video skims being the video summary.

8. A computer-readable memory comprising computer-program instructions executable by a processor to generate a video summary of a video data sequence, the computer-program instructions comprising instructions for:

identifying key-frames of the video data sequence;

creating a static summary of shots from the video data sequence according to criteria based on relative key-frame importance values; and

calculating one or more dynamic video skims for each shot in the static summary, the one or more dynamic skims being calculated based on a skim ratio for the shot, the video summary being the dynamic skims.

9. A computer-readable medium as recited in claim 8, wherein identifying the key frames is accomplished independent of shot boundary determinations.

11. A computer-readable medium as recited in claim 8, wherein the computer-executable instructions for identifying the key frames further comprise instructions for:

generating a derivative data curve from a comprehensive user attention model, the comprehensive user attention model being based at least on integrated visual and audio attention models; and

determining the key frames from respective peak attention values of the derivative data curve.

12. A computer-readable medium as recited in claim 8, wherein the computer-program instructions for creating further comprise instructions for:

evaluating whether a total number of shots in the static summary is more that a desired number of shots;

responsive to identifying that the total number of shots in the static summary is more that the desired number of shots;

determining that a shot is of lower importance than a different shot, the video data sequence comprising the shot and the different shot; and

dropping shots of lower importance as compared to shots of higher importance from the static summary.

13. A computer-readable medium as recited in claim 8, wherein the criteria comprise computer-executable instructions for:

responsive to determining that there are more key-frames than a threshold number of allowed shots;

assigning each shot in the video data sequence an importance value based on maximum key-frame importance in the shot, a shot without a key-frame being assigned a lowest importance value;

dropping shots with importance values that are low as compared to respective importance values of other shots; and

selecting only non-dropped shots for the static summary.

14. A computer-readable medium as recited in claim 8, wherein the instructions for calculating further comprise instructions for adjusting dynamic skim segment boundaries according to a sentence boundary.

15. A computer-readable medium as recited in claim 8, wherein the instructions for calculating further comprise instructions for, given a skim ratio for the shot, creating each of the one or more dynamic video skims with a respective length that is based on length of the shot and number of key-frames in the shot.

16. A computer-readable medium as recited in claim 8, wherein the instructions for calculating further comprise instructions for:

identifying a minimum number of frames for a dynamic skim;

determining that an average frame length of dynamic skims in the shot is less that the minimum number of frames; and

evenly distributing dynamic skim segment boundaries across the shot until an average frame length of dynamic skims in the shot is greater than the minimum number of frames.

17. A computing device to generate a video summary of a video data sequence, the video data sequence comprising multiple shots, the computing device comprising:

a processor; and

a memory coupled to the processor, the memory comprising computer-program instructions executable by the processor for;

identifying key-frames of the video data sequence;

calculating both static and dynamic summarization data from the key-frames; and

generating the video summary from the static and dynamic summarization data.

18. A computing device as recited in claim 17, wherein the static summarization data comprises a multi-scale static shot summary.

19. A computing device as recited in claim 17, wherein the dynamic summarization data comprises multiple dynamic video skims based on a static shot summary.

20. A computing device as recited in claim 17, wherein the comprehensive user attention model is an integrated set of multiple visual, audio, and linguistic attention model data.

21. A computing device as recited in claim 17, wherein the computer-program instructions for calculating further comprise instructions for generating the static summarization data based on relative key-frame importance values determined from a comprehensive user attention model of the video data sequence.

22. A computing device as recited in claim 17, wherein the computer-program instructions for calculating further comprise instructions for generating the dynamic summarization data by adjusting dynamic skim segment boundaries according to sentence boundaries.

23. A computing device as recited in claim 17, wherein the computer-program instructions for calculating further comprise instructions for generating the dynamic summarization data by:

identifying a minimum number of frames for a dynamic skim;

determining that an average frame length of dynamic skims in a shot identified in the static summarization data is less that the minimum number of frames; and

evenly distributing dynamic skim segment boundaries across the shot until an average frame length of dynamic skims in the shot is greater than the minimum number of frames.

24. A computing device to generate a video summary of a video data sequence, the video data sequence comprising multiple shots, the computing device comprising:

means for identifying, independent of shot boundary detection, key-frames of the video data sequence; and

means for calculating both static and dynamic summarization data from the key-frames, the video summary being based on the static and dynamic summarization data.

25. A computing device as recited in claim 24, wherein the static summarization data comprises a static shot summary calculated according to key-frame attention values.

26. A computing device as recited in claim 24, wherein the dynamic summarization data comprises multiple dynamic video skims determined according to shot and sentence boundaries.