Interactive custom video creation system

• US 6,807,361 B1
• Filed: 07/18/2000
• Issued: 10/19/2004
• Est. Priority Date: 07/18/2000
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Girgensohn, Andreas, Chiu, Patrick, Foote, Jonathan T., Boreczky, John S., Wilcox, Lynn D., Doherty, John J.
• Primary Examiner(s): Chevalier, Robert
• Application Number:
  US09/618,533
• Time in Patent Office: 1,554 Days
• Field of Search: H04/N.593, H04/N.576, G11/B 2.700, 386/52, 386/55, 386/53, 386/46, 386/107, 386/117, 386/124, 386/1, 386/4, 386/6, 386/68, 386/69, 360/13, 358/909.1, 348/222.1, 345/723
• US Class Current: 386/227
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

The present invention analyzes recorded video from a video camera to identify camera and object motion in the recorded video. Keyframes representative of clips of the recorded video are displayed on a user interface that allows a user to manipulate an order of the keyframes. Editing rules are then applied to the keyframes to intelligently splice together portions of the representative clips into a final output video.

Claims

1. A video creation system for analyzing recorded video from a video camera to produce a final output video, the system comprising:

a camera motion detector, the camera motion detector segmenting respective takes of the recorded video into clips, the clips being classified according to camera motion detected in respective takes; and

a video splicer, the video splicer applying a plurality of editing rules to determine whether video frames adjacent to keyframes representative of respective clips are spliced together to create the final output video.

2. The system of claim 1 further comprising a digitizer, the digitizer digitizing the recorded video and segmenting the recorded video into takes corresponding to camera on and camera off transitions.

3. The system of claim 1 further comprising a user interface, the user interface displaying respective selected keyframes and permitting a user to change a sequence of the keyframes to any desired order.

4. The system of claim 3, wherein the user interface comprises a keyframe interface and a storyboard interface.

5. The system of claim 4, wherein a size of respective keyframes displayed in the keyframe interface is proportional to a duration of respective clips represented by the corresponding keyframe.

6. The system of claim 4, wherein a user can arrange respective keyframes displayed in the keyframe interface onto the storyboard interface in any desired time sequence.

7. The system of claim 3, wherein the user interface is a dynamic user interface, the dynamic user interface enabling a user to omit the clip being viewed from the final output video while the recorded video is playing.

8. The system of claim 7, wherein the dynamic user interface is used in conjunction with a static user interface.

9. The system of claim 3, wherein the user interface enables a user to include the clip being viewed in the final output video.

10. The system of claim 1 further comprising a keyframe selector, the keyframe selector enabling a user to select at least one keyframe representative of respective clips.

11. The system of claim 10, wherein a single keyframe is selected and displayed for clips classified as a still class having a static scene.

12. The system of claim 10, wherein multiple keyframes are selected and displayed for clips having object motion therein.

13. The system of claim 12, wherein multiple keyframes are selected using hierarchical agglomerative clustering to segment respective clips into homogeneous regions and choosing keyframes from respective regions.

14. The system of claim 1, wherein the video splicer employs a constraint satisfaction system for applying the plurality of editing rules.

15. The system of claim 1, wherein an ergodic Hidden Markov Model is used to segment respective takes into clips based on camera motion classes detected in the take by the camera motion detector.

16. The system of claim 1, wherein respective clips are classified according to camera motion comprising a still class, a pan class, a tilt class, a zoom class, and a garbage class.

17. The system of claim 1, wherein the editing rules comprise:

discarding respective clips having a length less than a minimum length, the minimum length being substantially equal to three seconds;

trimming respective clips having a length that exceeds a maximum length, the maximum length being substantially equal to ten seconds;

merging two clips, to be included in the final output video, selected from the same take if the two clips are separated by less than three seconds to avoid cutting between the two clips;

discarding clips having fast and non-linear camera motion and clips classified as a garbage class;

selecting a sub-clip near an end of a shot if the shot exceeds the maximum length;

discarding clips comprising a zoom, a pan, and a tilt having durations less than five seconds unless a still clip exists on either end of the zoom, the pan, and the tilt; and

selecting the shot if the shot has a minimum brightness above a predetermined brightness threshold.

18. The system of claim 17, wherein the predetermined brightness threshold is substantially equal to 30% brightness.

19. The system of claim 1, wherein the video splicer applies the editing rules to automatically determine a video in point and a video out point for respective clips.

20. A method for creating custom videos from recorded video of a video camera, the method comprising the steps of:

detecting camera movement in respective takes of the recorded video;

segmenting respective takes into clips based on classes of camera movement detected;

displaying selected keyframes representative of respective clips on a user interface, the selection of respective keyframes based on the classes of camera movement detected; and

applying editing rules to the keyframes displayed in the user interface to choose sections of video around respective keyframes to splice together to create the final output video.

21. The method of claim 20 further comprising the step of digitizing the recorded video and segmenting the digitized recorded video into takes.

22. The method of claim 21, wherein respective takes are defined by digitized recorded video between a camera on transition and a camera off transition.

23. The method of claim 20, wherein an ergodic Hidden Markov Model segments respective takes into clips based on camera motion classes detected in the take by the camera motion detector.

24. The method of claim 23, wherein the camera motion classes comprise a still class, a pan class, a tilt class, a zoom class, and a garbage class.

25. The method of claim 20, wherein a single keyframe is selected for clips comprising camera motion classified as a still class having a static scene.

26. The method of claim 20, wherein multiple keyframes are selected for clips comprising camera motion respectively classified as a pan class, a tilt class, a zoom class, and a still class having object motion.

27. The method of claim 26, wherein multiple keyframes are selected using hierarchical agglomerative clustering to divide respective clips into homogeneous regions and choosing keyframes from respective regions.

28. The method of claim 20, wherein a size of respective selected keyframes displayed on the user interface is proportional to a duration of clips represented by respective selected keyframes.

29. The method of claim 28, wherein the user interface comprises a keyframe interface and a storyboard interface where a user can place respective selected keyframes displayed in the keyframe interface onto the storyboard interface in any desired sequence.

30. The method of claim 20, wherein the user interface comprises a first display interface and a second display interface.

31. The method of claim 30, wherein the static user interface displays selected keyframes and the dynamic user interface enables a user to dynamically select clips to be omitted from the final output video while the recorded video is playing.

32. The method of claim 20, wherein the step of applying editing rules further comprises the steps of:

discarding respective clips having a length less than a minimum length, the minimum length being substantially equal to three seconds;

trimming respective clips having a length that exceeds a maximum length, the maximum length being substantially equal to ten seconds;

merging two clips, to be included in the final output video, selected from the same take if the two clips are separated by less than three seconds to avoid cutting between the two clips;

discarding clips having fast and non-linear camera motion and clips classified as a garbage class;

selecting a sub-clip near an end of a shot if the shot exceeds the maximum length;

discarding clips comprising a zoom, a pan, and a tilt having durations less than five seconds unless a still clip exists on either end of the zoom, the pan, and the tilt; and

selecting the shot if the shot has a minimum brightness above a predetermined brightness threshold.

33. The method of claim 20, wherein the step of applying editing rules further comprises the step of automatically determining a video in point and a video out point for respective clips.

34. The method of claim 20, wherein the segmenting step segments respective takes into clips based, alternatively, on video quality rules.

35. The method of claim 34, wherein keyframes representative of clips having a highest video quality from respective takes are selected.