Data-Driven Video Stabilization

• US 20100165123A1
• Filed: 12/29/2008
• Published: 07/01/2010
• Est. Priority Date: 12/29/2008
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Matsushita, Yasuyuki, Wei, Yichen
• Application Number:
  US12/344,599
• Publication Number:
  US 20100165123A1
• Time in Patent Office: Days
• US Class Current: 348/208.400
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/003 Deblurring; Sharpening vibration or motion blur correction for cameras comprising an electronic image sensor H04N23/682
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations

Abstract

Described is a technology in which existing motion information (e.g., obtained from professional quality videos) is used to correct the motion information of an input video, such as to stabilize the input video. The input video is processed into an original motion chain, which is segmented into original segments. Candidate segments are found for each original segment, and one candidate segment is matched (based on matching criteria or the like) to each original segments. The matched candidates are stitched together to form a changed motion chain that via image warping changes the motion in the output video. Also described is building the data store by processing reference videos into motion information; different data stores may be built based upon styles of reference videos that match a particular style of motion (e.g., action video, scenic video) for the data store.

Claims

1. In a computing environment, a method comprising, inputting a video comprising a set of frames, determining existing motion information between each of the frames, locating other motion information from a data store of motion information, applying the other motion information to change the existing motion information between each of the frames into changed motion information, and outputting a changed video having the changed motion information.

2. The method of claim 1 wherein the existing motion information between each of the frames comprises a motion chain, and further comprising dividing the motion chain into a plurality of original segments.

3. The method of claim 2 wherein locating the other motion information comprises finding candidate segments for at least some of the original segments, and selecting corresponding segments from the candidate segments.

4. The method of claim 1 wherein applying the other motion information comprises performing image warping based on the changed motion information to transform data in one frame into transformed data in a subsequent frame.

5. The method of claim 1 wherein locating the other motion information comprises constraining the other motion information so as to have affinity with the original motion information.

6. The method of claim 1 wherein locating the other motion information comprises constraining the other motion information so as to be close in continuity and smoothness with the original motion information.

7. The method of claim 1 further comprising, building the data store, including processing reference videos into motion information, dividing, pruning or refining the motion information, or any combination of dividing, pruning or refining the motion information into sampled motion information, and maintaining the sampled motion information in the data store.

8. The method of claim 7 wherein building the data store comprises using reference videos that match a particular style of motion for the data store.

9. In a computing environment, a system comprising, a data store of predetermined motion information, a motion estimation mechanism that processes frames of input video into existing motion information for that input video, a motion synthesis mechanism coupled to the data store that selects sets of predetermined motion information corresponding at least some of the existing motion information, and an image warping mechanism that modifies at least some of the existing motion information with corresponding predetermined motion information as selected by the motion synthesis mechanism into modified motion information.

10. The system of claim 9 wherein the existing motion information between each of the frames comprises a motion chain, and wherein the motion synthesis mechanism divides the motion chain into a plurality of original segments, finds candidate segments for at least some of the original segments, and selects corresponding segments from the candidate segments.

11. The system of claim 9 wherein the image warping mechanism transforms data in one frame into transformed data in a subsequent frame based on the modified motion information.

12. The system of claim 9 wherein the motion synthesis mechanism selects the sets of predetermined motion information within one or more constraints.

13. The system of claim 12 wherein the one or more constraints include an affinity constraint, or a smoothness constraint, or both an affinity constraint and a smoothness constraint.

14. The system of claim 9 further comprising, means for building the data store, including a global estimation mechanism that processes reference videos into motion information, and a motion collection mechanism that divides, prunes and/or refines the motion information into the predetermined motion information.

15. The system of claim 9 wherein the data store corresponds to a style of motion information that is common among a set of reference videos used to build the data store.

16. One or more computer-readable media having computer-executable instructions, which when executed perform steps, comprising:

inputting a video comprising a set of frames;

determining an original motion chain between each of the frames;

dividing the motion chain into a plurality of original segments, each segment having associated motion information;

locating predetermined motion information for each original segment based upon the motion information corresponding to that segment;

applying the predetermined motion information to change the associated motion information of each segment into changed motion information; and

outputting a changed video having the changed motion information.

17. The one or more computer-readable media of claim 16 wherein applying the other motion information comprises performing image warping based on the changed motion information to transform data in one frame into transformed data in a subsequent frame.

18. The one or more computer-readable media of claim 16 wherein locating the other motion information comprises finding candidates for each segment, and selecting a candidate based upon matching at least one criterion.

19. The one or more computer-readable media of claim 16 wherein locating the other motion information comprises following at least one constraint.

20. The one or more computer-readable media of claim 16 having further computer-executable instructions comprising building the data store, including processing reference videos.