Extracting key frame candidates from video clip

• US 20070183497A1
• Filed: 02/03/2006
• Published: 08/09/2007
• Est. Priority Date: 02/03/2006
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Eastman Kodak Company
• Inventors: Luo, Jiebo, Papin, Christophe
• Granted Patent: US 7,889,794 B2
• Time in Patent Office: Days
• US Class Current: 375/240.120
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

A method of analyzing a digital video clip to determine candidate frames for subsequent key frame selection including performing a global motion estimate on the video clip that indicates translation of the scene or camera, or scaling of the scene; forming a plurality of video segments based on the global motion estimate and labeling each segment in accordance with a predetermined series of camera motion classes; extracting key frame candidates from the labeled segments and computing a confidence score for each candidate by using rules corresponding to each camera motion class and a rule corresponding to object motion.

Claims

1. A method of analyzing a digital video clip to determine candidate frames for subsequent key frame selection, comprising:

a. performing a global motion estimate on the video clip that indicates translation of the scene or camera, or scaling of the scene;

b. forming a plurality of video segments based on the global motion estimate and labeling each segment in accordance with a predetermined series of camera motion classes; and

c. extracting key frame candidates from the labeled segments and computing a confidence score for each candidate by using rules corresponding to each camera motion class and a rule corresponding to object motion.

2. The method of claim 1 wherein the predetermined camera motion classes include pan (left or right, and tilt up or down), zoom (in or out), fast pan or fixed.

3. The method of claim 2 wherein the rules include a pan rule, a zoom rule, a fast pan rule and a fixed rule.

4. The method of claim 3 wherein the pan rule includes extracting a plurality of frames to cover the space of environment while reducing the spatial overlap among the frames from a pan segment.

5. The method of claim 3 wherein the pan rule includes extracting a frame located at a point when the pan motion is slowed down.

6. The method of claim 3 wherein the zoom rule includes extracting a candidate frame at an endpoint of the zoom-in or zoom-out segment.

7. The method of claim 3 wherein the fast pan rule includes not extracting a candidate frame from a fast pan segment.

8. The method of claim 3 wherein the fixed rule includes extracting a candidate frame located at a midpoint of the fixed segment.

9. The method of claim 1 wherein the object motion rule includes extracting a candidate frame for a fixed segment with a confidence score related to the segment length, extracting a candidate frame for a pan segment with a confidence score related to a translation amount during the pan, and does not extract candidate frames for object motion based frames for fast pan and zoom segments.

10. A method of analyzing a digital video clip to determine key frames, comprising:

a. performing a global motion estimate on the video clip that indicates translation of the scene or camera, or scaling of the scene;

b. forming a plurality of video segments based on the global motion estimate and labeling each segment in accordance with a predetermined series of classes;

c. extracting key frame candidates from the labeled segments and computing a confidence score for each candidate by using rules associated with each label; and

d. selecting key frames from the candidate frames based on the confidence score of each candidate.

11. The method of claim 10 further including ranking the selected key frames in accordance with the confidence score.

12. The method of claim 11 wherein step d includes employing the ranking and a user specified number to select the key frames.

13. The method of claim 12 wherein employing the ranking and a user specified number to select the key frames includes selecting at least one key frame from each segment if there are confidence scores above a pre-determined threshold.

14. The method of claim 13 wherein employing the ranking and a user specified number to select the key frames includes selecting key frames from the remaining candidates with the highest confidence values to fill the user specified number of key frames.

15. The method of claim 10 wherein the predetermined camera motion classes include pan (left or right, and tilt up or down), zoom (in or out), fast pan or fixed.

16. The method of claim 15 wherein the rules include a pan rule, a zoom rule, a fast pan rule and a fixed rule.

17. The method of claim 16 wherein the pan rule includes extracting a plurality of frames to cover the space of environment while reducing the spatial overlap among the frames from a pan segment.

18. The method of claim 16 wherein the pan rule includes extracting a frame located at a point when the pan motion is slowed down.

19. The method of claim 16 wherein the zoom rule includes extracting a candidate frame at an endpoint of the zoom-in or zoom-out segment.

20. The method of claim 16 wherein the fast pan rule includes does not extract candidate frames from a fast pan segment.

21. The method of claim 16 wherein the fixed rule includes extracting a candidate frame located at a midpoint of the fixed segment.

22. The method of claim 10 wherein the object motion rule includes extracting a candidate frame for a fixed segment with a confidence score related to the segment length, extracting a candidate frame for a pan segment with a confidence score related to a translation amount during the pan, and does not extract candidate frames for object motion based frames for fast pan and zoom segments.