REAL TIME VIDEO STABILIZATION

• US 20120120265A1
• Filed: 06/22/2010
• Published: 05/17/2012
• Est. Priority Date: 06/22/2009
• Status: Active Grant

• Current Assignee: IMINT Image Intelligence AB
• Original Assignee: IMINT Image Intelligence AB
• Inventors: Klomp, Harald, Frisk, Anders
• Granted Patent: US 8,675,081 B2
• Time in Patent Office: Days
• US Class Current: 348/208.6
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations

Abstract

The present invention provides a method and a system for video stabilization in real time. The method comprises the steps of: estimating an optical flow from an previous image, I_i, to a current image, I_i+1; determining an Euclidean transformation, F_i, from the previous image, I_i to the current image, I_i+1, wherein the Euclidean transformation, F_i, comprises a product of rotation, R_i, and scaling, si, and translation, T_i; and transforming the current image, I_i+1, by applying the inverse of the Euclidean transformation, F_i, to the current image, I_i+1 in order to obtain stabilization. The transformation of the current image, I_i+1, is restricted by using Euclidean matrices A_i and B_i, wherein: Ai is used to store absolute frame transformation relative world frame coordinates, A₀ is the identity matrix and A_i+1 is equal to AiFi⁻¹ for i≧0 and B_i is used to store visualization window coordinate transformation.

Claims

1. A method for video stabilization of a sequence of images captured by a camera, the sequence of images comprising at least one successive image pair with a current image, I_i+1, and a previous image, I_i, i≧

0 denoting an image frame number, and the method comprising the steps of;

estimating an optical flow from the previous image, I_i, to the current image, I_i+1;

determining an Euclidean transformation, F_i, from the previous image, I_i, to the current image, I_i+1, wherein the Euclidean transformation, F_i, comprises a product of at least a rotation matrix, R_i, and a scaling factor, s_i, and a translation matrix, T_i;

restricting the transformation of the current image, I_i+1, by using Euclidean matrices A_i and B_i, wherein;

A_i is used to store absolute frame transformation relative world frame coordinates, A₀ is the identity matrix and A_i+1 is equal to A_iF_i<sup>−

1</sup> for i≧

0 and B_i is used to store visualization window coordinate transformation;

transforming the current image, I_i+1, by applying the inverse of the Euclidean transformation, F_i, to the current image, I_i+1.

2. The method according to claim 1, wherein F_i is a matrix of order three, and the F_i, comprises a product of a rotation matrix, R_i, and a scaling factor, s_i, and a translation matrix, T_i.

3. The method according to claim 1 or 2, wherein the step of estimating comprises the sub-steps of:

selecting tie points in the current image, I_i+1, and the previous image, I_i; and

tracking corresponding tie points in the current image, I_i+1, and the previous image, I_i.

4. The method according to claim 1, wherein B_i is set to the identity matrix to obtain full stabilization.

5. The method according to claim 1, wherein the step of restricting comprises, in order to calculate B_i+1, one or a combination of the sub-steps of:

calculating translation x_i+1 as follows;

x_i+1=u_i−

Sign(x)R if |x|>

R_x, or else x_i+1=x_i+α

(x,F_x,R_x)C_x, wherein u_i is a x-translation given by A_i, x=u_i−

x_i, R_x is a predetermined constant determining the maximum difference in x-position between the transformed image and the camera, α

is a function of x, F_x, and R_xF_x is a predetermined percentage of R_x defining maximum stabilization, and C_x is a predetermined constant defining the effect of stabilization;

calculating translation y_i+1 as follows;

y_i+1=v_i−

Sign(y)R if |y|>

R_y, or else y_i+1=y_i+α

(y,F_y,R_y)C_y, wherein y_i is a y-translation given by A_i, y=v_i−

y_i, R_y is a predetermined constant determining the maximum difference in y-position between the transformed image and the camera, α

is a function of y, F_y, and R_y, F_y is a predetermined percentage of R_y defining maximum stabilization, and C_y is a predetermined constant defining the effect of stabilization; and

calculating rotation angle θ

_i+1 as follows;

θ

_{i i+1}=w_i−

Sign(θ

_i)R_θ

if |θ

_i|>

R_θ

, or else θ

_i+1=θ

_i+α

(θ

,F,R_θ

)C_θ

, wherein w_i is a θ

-rotation given by A_i, θ

=w_i−

θ

_i converted to the interval [−

π

,π

], R_θ

is a predetermined constant determining the maximum difference in θ

-rotation between the transformed image and the camera, α

is a function of θ

, F_θ

, and R_θ

, F_θ

, is a predetermined percentage of R_θ

defining maximum stabilization, and C_θ

is a predetermined constant defining the effect of stabilization.

6. The method according to claim 5, wherein the transformation of the current image is progressive in order to perform larger stabilization for small variations and less stabilization for large variations.

7. The method according to claim 5, wherein the function α

is defined by α

(e,f,r)=0 if |e|<

fr or else

8. The method according to claim 1, further comprising the step of removing outliers.

9. A system for video stabilization of a sequence of images captured by a camera, the sequence of images comprising at least one successive image pair with a current image, I_i+1, and a previous image, I_i, i≧

0 denoting an image frame number, wherein the system comprises;

an estimation module adapted to for estimating the optical flow from the previous image, I_i, to the current image, I_i+1,

a calculation module adapted to determine an Euclidean transformation matrix, F_i, of order three from the previous image, I_i, to the current image, I_i+1;

a transformation module adapted to transform the current image, I_i+1 by applying the Euclidean transformation matrix, to obtain stabilization.