Method for determining key video frames

• US 8,599,316 B2
• Filed: 05/25/2010
• Issued: 12/03/2013
• Est. Priority Date: 05/25/2010
• Status: Expired due to Fees

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Intellectual Ventures Fund 83 LLC (Intellectual Ventures LLC)
• Inventors: Deever, Aaron T.
• Primary Examiner(s): Zhao, Daquan
• Assistant Examiner(s): DUNN, MISHAWN N
• Application Number:
  US12/786,471
• Publication Number:
  US 20110292288A1
• Time in Patent Office: 1,288 Days
• Field of Search: 386/241, 386/242, 348/700, 348/590, 382/173, 382/181, 382/190, 382224-228
• US Class Current: 348/590
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: H04N 5/275 Generation of keying signals

Abstract

A method for determining key video frames for a digital video having a time sequence of video frames, comprising using a processor to perform the steps of: analyzing the digital video to determine an importance value as a function of time; forming a warped time representation by temporal relocation of the video frames responsive to the importance value as a function of time; subdividing the warped time digital video sequence representation into a set of equal intervals; selecting a key video frame for each interval by analyzing the video frames within each interval; and storing an indication of the key video frames in a processor-accessible memory.

Claims

1. A method comprising:

determining, by a computing device, an importance value for each of a plurality of first video intervals comprising a video frame, wherein a video comprises the plurality of first video intervals in a temporal order;

forming, by the computing device, a warped time representation of the video by modifying a duration of at least one of the first video intervals based at least in part on the importance values;

subdividing, by the computing device, the warped time representation of the video into second video intervals;

selecting, by the computing device, a key video frame for each second video interval; and

storing, by the computing device, an indication of the key video frame for each second video interval.

2. The method of claim 1, wherein determining the importance value comprises detecting global motion.

3. The method of claim 1, wherein determining the importance value comprises detecting local motion.

4. The method of claim 1, wherein determining the importance value comprises classifying the video frame into at least one of a plurality of video frame classifications.

5. The method of claim 4, wherein the plurality of video frame classifications include zoom, fast pan, interesting, or inactive.

6. The method of claim 4, wherein each of the plurality of video frame classifications is associated with an importance value.

7. The method of claim 1, wherein selecting the key video frames comprises determining a selection score for each of a plurality of second video interval frames within each second video interval.

8. The method of claim 7, wherein determining the selection scores is based at least part on global motion and local motion.

9. The method of claim 1, wherein selecting the key video frames comprises selecting an independently encoded video frame as the key video frame within each second video interval.

10. The method of claim 1, wherein storing the indication of the key video frames comprises extracting the key video frames as separate files.

11. The method of claim 1, wherein storing the indication of the key video frames comprises storing a list of key video frames as metadata associated with the video.

12. The method of claim 1, wherein storing the indication of the key video frames comprises storing thumbnail images for the key video frames as metadata associated with the video.

13. The method of claim 1, wherein forming the warped time representation of the video comprises assigning a shorter temporal extent to video frames having importance values indicating lower importance.

14. The method of claim 1, wherein forming the warped time representation of the video comprises assigning a longer temporal extent to video frames having importance values indicating higher importance.

15. The method of claim 1, wherein the number of second video intervals is user-specified.

16. The method of claim 1, further comprising:

ranking, by the computing device, the key video frames based at least in part on a criterion;

forming, by the computing device, key video snippets corresponding to the highest-ranked key video frames;

forming, by the computing device, a video summary by assembling the key video snippets; and

storing, by the computing device, a representation of the video summary.

18. The method of claim 1, wherein the temporal order of the plurality of first video intervals is maintained in the warped time representation of the video.

17. A system comprising:

an image sensor;

an optical system configured to form an image of a scene onto the image sensor;

a data processing system; and

a memory system communicatively connected to the data processing system and storing instructions configured to cause the data processing system to;

capture a video using the image sensor, wherein the video comprises a plurality of first video intervals in a temporal order, wherein each of the plurality of first video intervals comprises a video frame;

determine an importance value for each of the plurality of first video intervals comprising a video frame;

form a warped time representation by modifying a duration of at least one of the first video intervals based at least in part on the importance values without modifying the temporal order of the video frames;

subdivide the warped time representation into intervals;

select a key video frame for each interval; and

store an indication of the key video frames.