Determining key video snippets using selection criteria to form a video summary

• US 8,605,221 B2
• Filed: 05/25/2010
• Issued: 12/10/2013
• Est. Priority Date: 05/25/2010
• Status: Expired due to Fees

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Intellectual Ventures Fund 83 LLC (Intellectual Ventures LLC)
• Inventors: Deever, Aaron T.
• Primary Examiner(s): Zhao, Daquan
• Assistant Examiner(s): DUNN, MISHAWN N
• Application Number:
  US12/786,480
• Publication Number:
  US 20110293250A1
• Time in Patent Office: 1,295 Days
• Field of Search: 386/224, 386/240, 386/241, 386/278, 386/329, 386/343, 386/239, 386/248, 348/207.99, 348/231.2, 348/590, 348/700, 382/162, 382/168, 382/190, 382/225, 375/240.01, 375/240.08, 375/240.12, 375/240.13
• US Class Current: 348/590
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: H04N 5/275 Generation of keying signals

Abstract

A method for forming a video summary for a digital video having a time sequence of video frames, comprising using a processor to perform the steps of: determining a set of key video frames for the digital video; ranking the key video frames responsive to a predefined criterion; forming key video snippets corresponding to one or more highest-ranked key video frames according to a predefined set of criteria, including a criterion which specifies the total time duration for the video summary and a criterion which specifies the minimum time duration for each of the key video snippets; assembling the key video snippets to form a video summary; and storing a representation of the video summary in a processor-accessible memory.

Claims

1. A method for forming a video summary for a digital video having a time sequence of video frames, the method comprising:

determining, by a computing device, a set of video frames for the digital video;

ranking, by the computing device, the set of video frames;

forming, by the computing device, video snippets corresponding to one or more highest-ranked video frames according to a predefined set of criteria, wherein the predefined set of criteria comprises a criterion which specifies a total time duration for the video summary and a criterion which specifies a minimum time duration for each of the video snippets;

assembling, by the computing device, the video snippets to form the video summary;

storing, by the computing device, a representation of the video summary in a memory; and

merging video snippets corresponding to highest-ranked video frames that are separated by time intervals less than a specified threshold to form a single video snippet.

2. The method of claim 1, wherein each highest-ranked video frame is included in a video snippet.

3. The method of claim 1, further comprising analyzing the digital video to determine an importance value as a function of time, wherein start and end times for at least one of the video snippets are determined responsive to a criterion which encourages inclusion of video frames having high importance values.

4. The method of claim 1, further comprising analyzing the digital video to determine a level of audio activity as a function of time, wherein start and end times for at least one of the video snippets are determined responsive to a criterion which encourages selecting start and end times corresponding to low levels of audio activity.

5. The method of claim 1, wherein the forming the video snippets comprises forming the video snippets according to a criterion which discourages the formation of video snippets that have similar content.

6. The method of claim 1, wherein the total time duration for the video summary is user-specified.

7. The method of claim 1, wherein the determining a set of video frames for the digital video comprises:

analyzing the digital video to determine an importance value as a function of time;

forming a warped time representation by temporal relocation of the video frames responsive to the importance value as a function of time;

subdividing the warped time representation into a set of equal intervals; and

selecting a video frame for each interval by analyzing the video frames within each interval.

8. The method of claim 1, wherein the ranking the video frames responsive to a predefined criterion comprises:

analyzing the digital video to determine a camera fixation pattern;

ranking the video frames responsive to the camera fixation pattern.

12. The method of claim 1, wherein the video snippets are formed according to a criterion which favors starting a video snippet at a beginning of an encoding group of video frames and ending a video snippet at an end of the encoding group of video frames, wherein the encoding group of video frames includes an independently-encoded video frame and a plurality of predictive-encoded video frames.

13. The method of claim 1, further comprising classifying each video frame as a zoom video frame, a fast pan video frame, an inactive video frame, or an interesting video frame, wherein each of the zoom video frame, the fast pan video frame, or the interesting video frame have a different ranking value.

14. The method of claim 13, wherein the zoom video frame comprises a video frame where a camera is zooming in or out while the video frame is being captured, wherein the fast pan video frame comprises a video frame having a magnitude of global translational motion occurring that exceeds a threshold, wherein the inactive video frame comprises a video frame having a magnitude of global translational motion occurring that is below a threshold, and wherein the interesting video frame comprises all video frames not classified as the zoom video frame, the fast pan video frame, or the inactive video frame.

9. A digital video camera system comprising:

an image sensor;

an optical system configured to form an image of a scene onto the image sensor;

a data processing system; and

a memory system communicatively connected to the data processing system and storing instructions configured to cause the data processing system to implement a method for forming a video summary for a digital video, wherein the instructions comprise;

instructions to capture a digital video using the image sensor, the digital video having a time sequence of video frames;

instructions to determine a set of video frames for the digital video;

instructions to rank the video frames;

instructions to form video snippets corresponding to one or more highest-ranked video frames according to a predefined set of criteria, wherein the predefined set of criteria comprises a criterion which specifies a total time duration for a video summary and a criterion which specifies a minimum time duration for each of the video snippets;

instructions to assemble the video snippets to form the video summary;

instructions to store a representation of the video summary in a memory; and

instructions to merge video snippets corresponding to highest-ranked video frames that are separated by time intervals less than a specified threshold to form a single video snippet.

15. The digital video camera system of claim 9, wherein the memory system further stores instructions to merge video snippets corresponding to highest-ranked video frames that are separated by time intervals less than a specified threshold to form a single video snippet.

16. The digital video camera system of claim 9, wherein the instructions to determine a set of video frames for the digital video comprises:

instructions to analyze the digital video to determine an importance value as a function of time;

instructions to form a warped time representation by temporal relocation of the video frames responsive to the importance value as a function of time;

instructions to subdivide the warped time representation into a set of equal intervals; and

instructions to select a video frame for each interval by analyzing the video frames within each interval.

10. A method for forming a video summary for a digital video having a time sequence of video frames, the method comprising:

determining, by a computing device, a set of video frames for the digital video;

ranking, by the computing device, the video frames;

forming, by the computing device, video snippets corresponding to one or more highest-ranked video frames according to a predefined set of criteria, wherein the predefined set of criteria comprises a criterion which specifies a total time duration for the video summary and a criterion which specifies a minimum time duration for each of the video snippets, and wherein the forming video snippets comprises merging video snippets corresponding to highest-ranked video frames that are separated by time intervals less than a specified threshold to form a single video snippet;

assembling, by the computing device, the video snippets to form the video summary; and

storing, by the computing device, a representation of the video summary in a memory.

11. A method for forming a video summary for a digital video having a time sequence of video frames, the method comprising:

determining, by a computing device, a set of video frames for the digital video, wherein the determining comprises;

analyzing the digital video to determine an importance value as a function of time;

forming a warped time representation by temporal relocation of the set of video frames responsive to the importance value as a function of time;

subdividing the warped time representation into a set of equal intervals; and

selecting a video frame for each interval by analyzing the video frames within each interval;

ranking, by the computing device, the video frames;

forming, by the computing device, video snippets corresponding to one or more highest-ranked video frames according to a predefined set of criteria, wherein the predefined set of criteria comprises a criterion which specifies a total time duration for the video summary and a criterion which specifies a minimum time duration for each of the video snippets;

assembling, by the computing device, the video snippets to form the video summary; and

storing, by the computing device, a representation of the video summary in a memory.