Methods and Apparatus for Robust Video Stabilization

• US 20130128066A1
• Filed: 11/21/2011
• Published: 05/23/2013
• Est. Priority Date: 04/08/2011
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Jin, Hailin, Agarwala, Aseem O., Wang, Jue
• Granted Patent: US 8,724,854 B2
• Time in Patent Office: Days
• US Class Current: 348/208.99
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20028 Bilateral filtering
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/33 using feature-based methods

Abstract

Methods and apparatus for robust video stabilization. A video stabilization technique applies a feature tracking technique to an input video sequence to generate feature trajectories. The technique applies a video partitioning technique to segment the input video sequence into factorization windows and transition windows. The technique smoothes the trajectories in each of the windows, in sequence. For factorization windows, a subspace-based optimization technique may be used. For transition windows, a direct track optimization technique that uses a similarity motion model may be used. The technique then determines and applies warping models to the frames in the video sequence. In at least some embodiments, the warping models may include a content-preserving warping model, a homography model, a similarity transform model, and a whole-frame translation model. The warped frames may then be cropped according to a cropping technique.

Claims

1. A method, comprising:

performing, by one or more computers;

partitioning an input video sequence into one or more factorization windows and one or more transition windows according to two-dimensional (2D) feature trajectories in the video sequence, each window including one or more contiguous frames of the video sequence, wherein a factorization window includes frames to be processed by a factorization window stabilization technique, and wherein a transaction window includes frames to be processed by a transition window stabilization technique;

applying the factorization window stabilization technique to each of the one or more factorization windows;

applying the transition window stabilization technique to each of the one or more transition windows; and

applying one or more warping models to the frames in the video sequence according to the smoothed feature trajectories to generate a stabilized video sequence.

2. The method as recited in claim 1, wherein the factorization window stabilization technique and the transition window stabilization technique smooth the feature trajectories in the respective windows.

3. The method as recited in claim 1, wherein the factorization window stabilization technique and the transition window stabilization technique apply a no motion effect to the respective windows.

4. The method as recited in claim 1, wherein said partitioning the input video sequence into the one or more factorization windows and the one or more transition windows comprises sequentially processing the frames in the video sequence starting at the first frame of the video sequence and ending at the last frame of the video sequence to segment the video sequence into the two types of windows by iteratively performing:

sequentially examining each frame to determine if the frame can be processed by the factorization window stabilization technique until an initial frame is found that can be processed by the factorization window stabilization technique and assigning the initial frame to a new factorization window;

sequentially examining each frame in the video sequence after the initial frame to determine if the frame can be processed by the factorization window stabilization technique until a frame is found that cannot be processed by the factorization window stabilization technique, wherein any of the examined frames that can be processed by the factorization window stabilization technique are assigned to the current factorization window; and

if there are one or more unassigned frames prior to the initial frame, assigning the one or more frames to a new transition window.

5. The method as recited in claim 4, further comprising extending at least one transition window to overlap at least one adjacent factorization window by one or more frames.

6. The method as recited in claim 1, wherein said applying the factorization window stabilization technique to a factorization window comprises:

factoring the feature trajectories from the factorization window into a coefficient matrix representing features in the input video sequence and basis vectors representing camera motion over time in the input video sequence;

applying an energy minimization technique to the basis vectors to generate smoothed basis vectors; and

generating smoothed feature trajectories for the factorization window by combining the smoothed basis vectors with the coefficient matrix.

7. The method as recited in claim 5, wherein the energy minimization technique treats the basis vectors as unknowns in an optimization framework and computes a solution for each unknown in the optimization framework that minimizes an energy function that is the sum of a first data term that keeps the smoothed feature trajectories close in position to the original feature trajectories, a smoothness term that smoothes the feature trajectories over time, and a second data term that preserves temporal consistency between overlapping windows.

8. The method as recited in claim 1, wherein said applying the transition window stabilization technique to each of the one or more transition windows to smooth the feature trajectories in the one or more transition windows comprises applying a 2-D parametric transformation to each of the one or more transition windows.

9. The method as recited in claim 8, wherein the 2-D parametric transformation is a global 2-D similarity transformation.

10. The method as recited in claim 1, wherein said applying the transition window stabilization technique to each of the one or more transition windows to smooth the feature trajectories in the one or more transition windows comprises:

subdividing the transition window into one or more subwindows each including frames that do not include a sufficient number of features for applying a 2-D parametric transformation and one or more subwindows each including frames that do include a sufficient number of features for applying the 2-D parametric transformation;

for each of the one or more subwindows each including frames that do not include a sufficient number of features for applying the 2-D parametric transformation, setting each output feature trajectory transformation for the subwindow to the 2-D identity transformation; and

for each of the one or more subwindows each including frames that do include a sufficient number of features for applying the 2-D parametric transformation, applying the 2-D parametric transformation to the subwindow.

11. The method as recited in claim 1, wherein said applying one or more warping models to the frames in the video sequence comprises:

assigning an initial warping score to each frame, wherein the initial warping score indicates one of a plurality of warping models;

determining quality of each frame according to the smoothed feature trajectories for the respective frame;

adjusting the initial warping score for each frame according to the determined quality of the frame to generate an adjusted warping score for the respective frame; and

warping each frame according to one of the plurality of warping models indicated by the adjusted warping score for the respective frame.

12. The method as recited in claim 11, wherein the plurality of warping models include two or more of a content-preserving warping model, a homography model, a similarity transform model, or a whole-frame translation model.

13. The method as recited in claim 11, wherein for at least one frame said warping the frame comprises:

generating a first warped mesh for the frame according to a first warping model;

generating a second warped mesh for the frame according to a second warping model; and

warping the frame according to a combination of the first warped mesh and the second warped mesh.

14. The method as recited in claim 1, further comprising cropping the warped frames in the stabilized video sequence to generate an output video sequence.

15. The method as recited in claim 14, wherein said cropping the warped frames in the stabilized video sequence comprises:

determining an anchor point at each warped frame according to a maximum bounding box for the respective frame temporally smoothing the determined anchor points; and

cropping the warped frames according to the temporally smoothed anchor points.

16. A system, comprising:

at least one processor; and

a memory comprising program instructions, wherein the program instructions are executable by the at least one processor to;

partition an input video sequence into one or more factorization windows and one or more transition windows according to two-dimensional (2D) feature trajectories in the video sequence, each window including one or more contiguous frames of the video sequence, wherein a factorization window includes frames to be processed by a factorization window stabilization technique, and wherein a transaction window includes frames to be processed by a transition window stabilization technique;

apply the factorization window stabilization technique to each of the one or more factorization windows;

apply the transition window stabilization technique to each of the one or more transition windows; and

apply one or more warping models to the frames in the video sequence according to the smoothed feature trajectories to generate a stabilized video sequence.

17. The system as recited in claim 16, wherein, to apply the factorization window stabilization technique to a factorization window, the program instructions are executable by the at least one processor to:

factor the feature trajectories from the factorization window into a coefficient matrix representing features in the input video sequence and basis vectors representing camera motion over time in the input video sequence;

apply an energy minimization technique to the basis vectors to generate smoothed basis vectors, wherein the energy minimization technique treats the basis vectors as unknowns in an optimization framework and computes a solution for each unknown in the optimization framework that minimizes an energy function for the feature trajectories; and

generate smoothed feature trajectories for the factorization window by combining the smoothed basis vectors with the coefficient matrix.

18. The system as recited in claim 16, wherein, to apply the transition window stabilization technique to each of the one or more transition windows to smooth the feature trajectories in the one or more transition windows, the program instructions are executable by the at least one processor to apply a 2-D parametric transformation to each of the one or more transition windows.

19. A non-transitory computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to implement:

partitioning an input video sequence into one or more factorization windows and one or more transition windows according to two-dimensional (2D) feature trajectories in the video sequence, each window including one or more contiguous frames of the video sequence, wherein a factorization window includes frames to be processed by a factorization window stabilization technique, and wherein a transaction window includes frames to be processed by a transition window stabilization technique;

applying the factorization window stabilization technique to each of the one or more factorization windows;

applying the transition window stabilization technique to each of the one or more transition windows; and

applying one or more warping models to the frames in the video sequence according to the smoothed feature trajectories to generate a stabilized video sequence.

20. The non-transitory computer-readable storage medium as recited in claim 19, wherein, in said applying the factorization window stabilization technique to a factorization window, the program instructions are computer-executable to implement:

factoring the feature trajectories from the factorization window into a coefficient matrix representing features in the input video sequence and basis vectors representing camera motion over time in the input video sequence;

applying an energy minimization technique to the basis vectors to generate smoothed basis vectors, wherein the energy minimization technique treats the basis vectors as unknowns in an optimization framework and computes a solution for each unknown in the optimization framework that minimizes an energy function for the feature trajectories; and

generating smoothed feature trajectories for the factorization window by combining the smoothed basis vectors with the coefficient matrix.

21. The non-transitory computer-readable storage medium as recited in claim 19, wherein, in said applying the transition window stabilization technique to each of the one or more transition windows to smooth the feature trajectories in the one or more transition windows, the program instructions are computer-executable to implement applying a 2-D parametric transformation to each of the one or more transition windows.

22. The non-transitory computer-readable storage medium as recited in claim 19, wherein, in said applying one or more warping models to the frames in the video sequence, the program instructions are computer-executable to implement:

assigning an initial warping score to each frame, wherein the initial warping score indicates one of a plurality of warping models;

determining quality of each frame according to the smoothed feature trajectories for the respective frame;

adjusting the initial warping score for each frame according to the determined quality of the frame to generate an adjusted warping score for the respective frame; and

warping each frame according to one of the plurality of warping models indicated by the adjusted warping score for the respective frame.