Automatic alignment of video frames for image processing

• US 20080063298A1
• Filed: 09/10/2007
• Published: 03/13/2008
• Est. Priority Date: 09/13/2006
• Status: Active Grant

• Current Assignee: Providence Photonics, LLC (Metwood, Inc.)
• Original Assignee: Providence Engineering And Environmental Group, LLC
• Inventors: Zhou, Liming, Zeng, Yousheng
• Granted Patent: US 8,009,932 B2
• Time in Patent Office: Days
• US Class Current: 382/280
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 7/37 using transform domain methods
• CPC Class Codes: G06V 10/243 by compensating for image skew or non-uniform image deformations

Abstract

The provided is a method that can automatically align image frames in recorded video clips. Individual frames in video may shift and rotate due to shaking or vibration of a video camera. Unaligned frames make some imaging processing techniques difficult or infeasible. One example of image processing techniques is to isolate, recognize, and/or quantitatively analyze vapor plume images captured by an Infrared (IR) camera. Such techniques have a great potential to be used to automatically detect volatile organic compounds (VOC) leaked from process equipment at refineries and chemical plants. This invention is a technique for various subsequent image processing techniques. The invention uses spatially based Fast Fourier Transforms (FFT) to determine amount of shift, rotation, and scaling to align image frames, and uses a digital filtering technique to improve the alignment.

Claims

1. A method for aligning image frames, comprising steps of:

selecting a reference image;

selecting a sample image to be aligned to the reference image;

Fourier-transforming the reference image to obtain a Fourier transform of the reference image;

Fourier-transforming the sample image to obtain a Fourier transform of the sample image;

coordinate-transforming an absolute value of the Fourier transform of the reference image to obtain a coordinate-transformed Fourier transform of the reference image;

coordinate-transforming an absolute value of the Fourier transform of the sample image to obtain a coordinate-transformed Fourier transform of the sample image;

obtaining a first phase shift from the coordinate-transformed Fourier transform of the reference image and the coordinate-transformed Fourier transform of the sample image;

inverse-Fourier-transforming the first phase shift to obtain an inverse Fourier transform of the first phase shift;

finding a first transformation factor from the inverse Fourier transform of the first phase shift;

transforming the sample image by the first transformation factor to obtain a first-transformed sample image;

Fourier-transforming the first-transformed sample image to obtain a Fourier transform of the first-transformed sample image;

obtaining a second phase shift of the Fourier transform of the reference image and the Fourier transform of the first-transformed sample image;

inverse-Fourier-transforming the second phase shift to obtain an inverse Fourier transform of the second phase shift;

finding a second transformation factor from the inverse Fourier transform of the second phase shift; and

transforming the first-transformed sample image by the second transformation factor.

2. The method of claim 1, comprised of each of the reference image and the sample image being represented in Cartesian coordinates.

3. The method of claim 2, comprised of the step of coordinate-transforming the absolute value of the Fourier transform of the reference image including a step of transforming an absolute value of the Fourier transform of the reference image from Cartesian coordinates to log-polar coordinates;

and

comprised of the step of coordinate-transforming the absolute value of the Fourier transform of the sample image including a step of transforming an absolute value of the Fourier transform of the sample image from Cartesian coordinates to log-polar coordinates.

4. The method of claim 3, comprised of the first transformation factor including a rotational shift, by which the sample image is rotated to be aligned to the reference image.

5. The method of claim 4, comprised of the first transformation factor including a scaling factor, by which the sample image is rescaled to be aligned to the reference image.

6. The method of claim 4, comprised of the second transformation factor including a translational shift, by which the sample image is translated to be aligned to the reference image.

7. The method of claim 6, comprised of the step of finding the second transformation factor comprising steps of:

finding a first location (x₁, y₁) at which the inverse Fourier transform of the second phase shift is maximized;

finding a second location (x₂, y₂) among four points (x₁±

1, y₁±

1) at which the inverse Fourier transform of the second phase shift is the largest; and

finding the translational shift from formulas

$x_T=\frac{w_{x<br><br><br><br>1}<br><br>x_1+w_{x<br><br><br><br>2}<br><br>x_2}{w_{x<br><br><br><br>1}+w_{x<br><br><br><br>2}}$ $y_T=\frac{w_{y<br><br><br><br>1}<br><br>y_1+w_{y<br><br><br><br>2}<br><br>y_2}{w_{y<br><br><br><br>1}+w_{y<br><br><br><br>2}}$

where (x_T, y_T) is the translational shift, and w_xi and w_yi are defined by formulas

w_xi=∫

(|F(x_i,y₁)|)+∫

(|F(x_i,y₂)|)

w_yi=∫

(|F(x₁,y_i)|)+∫

(|F(x₂,y_i)|)

where i stands for 1 or 2, F stands for Fourier transform, and ∫

is an empirical function.

8. The method of claim 7, comprised of the empirical function including a function of ∫

(z)=z^α

, where α

is a parameter.

9. The method of claim 1, comprised of the step of finding a first transformation factor comprising steps of:

finding a first location (x₁, y₁) at which the inverse Fourier transform of the first phase shift is maximized; and

finding a second location (x₂, y₂) among four points (x₁±

1, y₁±

1) at which the inverse Fourier transform of the first phase shift is the largest.

10. The method of claim 1, comprised of the step of obtaining the first phase shift including a step of calculating the first phase shift from a formula

$R_1=\frac{F_1}{}$

( ξ

, η

) ·

conj 

( F 2 

( ξ

, η

) ) abs 

( F 1 

( ξ

, η

) ) ·

abs 

( F 2 

( ξ

, η

) ) ;

and

comprised of the step of obtaining the second phase shift including a step of calculating the second phase shift from a formula

$R_2=\frac{F_1<br><br>\left(\mathrm{\xi <br><br>},\mathrm{\eta <br><br>}\right)·<br><br>\mathrm{conj}<br><br>\left(F_2^{\mathrm{\prime <br><br>}}<br><br>\left(\mathrm{\xi <br><br>},\mathrm{\eta <br><br>}\right)\right)}{\mathrm{abs}<br><br>\left(F_1<br><br>\left(\mathrm{\xi <br><br>},\mathrm{\eta <br><br>}\right)\right)·<br><br>\mathrm{abs}<br><br>\left(F_2^{\mathrm{\prime <br><br>}}<br><br>\left(\mathrm{\xi <br><br>},\mathrm{\eta <br><br>}\right)\right)}$

where R₁ is the first phase shift, R₂ is the second phase shift, F₁(ξ

, η

) is the Fourier transform of the reference image, F₂(ξ

, η

) is the coordinate-transformed Fourier transform of the sample image, F₂′

(ξ

, η

) is the Fourier transform of the first-transformed sample image, ξ and

η

are a vertical and a horizontal frequencies, respectively, conj is a complex conjugate, and abs is an absolute value.

11. The method of claim 1, further comprising:

applying an erosion-dilation filter to process a difference image of the reference image and the transformed first-transformed sample image; and

reconstructing the transformed first-transformed sample image by adding the difference image to the reference image.

12. A machine readable storage medium providing instructions that cause the machine to perform operations to align image frames, the operations comprising:

selecting a reference image that is represented in Cartesian coordinates;

selecting a sample image to be aligned to the reference image, the sample image being represented in Cartesian coordinates;

Fourier-transforming the reference image to obtain a Fourier transform of the reference image;

Fourier-transforming the sample image to obtain a Fourier transform of the sample image;

coordinate-transforming an absolute value of the Fourier transform of the reference image from Cartesian coordinates to log-polar coordinates to obtain a coordinate-transformed Fourier transform of the reference image;

coordinate-transforming an absolute value of the Fourier transform of the sample image from Cartesian coordinates to log-polar coordinates to obtain a coordinate-transformed Fourier transform of the sample image;

obtaining a rotational phase shift from the coordinate-transformed Fourier transform of the reference image and the coordinate-transformed Fourier transform of the sample image;

inverse-Fourier-transforming the rotational phase shift to obtain an inverse Fourier transform of the rotational phase shift;

finding a rotational shift from the inverse Fourier transform of the rotational phase shift;

rotating the sample image by the rotational shift to obtain a rotated sample image;

Fourier-transforming the rotated sample image to obtain a Fourier transform of the rotated sample image;

obtaining a translational phase shift of the Fourier transform of the reference image and the Fourier transform of the rotated sample image;

inverse-Fourier-transforming the translational phase shift to obtain an inverse Fourier transform of the translational phase shift;

finding a translational shift from the inverse Fourier transform of the translational phase shift; and

translating the rotated sample image by the translational shift.

13. The machine readable storage medium of claim 12, the operations further comprising:

finding a scaling factor from the inverse Fourier transform of the rotational phase shift; and

rescaling the sample image by the scaling factor.

14. The machine readable storage medium of claim 12, wherein the step of obtaining the rotational phase shift includes a step of calculating the rotational phase shift from a formula

$R={}^{}$

- j 



2 



π





( ξ

·

dx + η

·

dy ) = Flp 1 

( ξ

, η

) ·

conj 

( Flp 2 

( ξ

, η

) ) abs 

( Flp 1 

( ξ

, η

) ) ·

abs 

( Flp 2 

( ξ

, η

) )

where R is the rotational phase shift, Flp₁(ξ

, η

) is the coordinate-transformed Fourier transform of the reference image, Flp₂(ξ

, η

) is the coordinate-transformed Fourier transform of the sample image, ξ and

η

are a vertical and a horizontal frequencies, respectively, conj is a complex conjugate, and abs is an absolute value.

15. The machine readable storage medium of claim 12, wherein the step of obtaining the translational phase shift includes a step of calculating the translational phase shift from a formula

$R={}^{}$

- j 



2 



π





( ξ

·

dx + η

·

dy ) = F 1 

( ξ

, η

) ·

conj 

( F 2 

( ξ

, η

) ) abs 

( F 1 

( ξ

, η

) ) ·

abs 

( F 2 

( ξ

, η

) )

where R is the translational phase shift, F₁(ξ

, η

) is the Fourier transform of the reference image, F₂(ξ

, η

) is the Fourier transform of the rotated sample image, ξ and

η

are a vertical and a horizontal frequencies, respectively, conj is a complex conjugate, and abs is an absolute value.

16. The machine readable storage medium of claim 12, wherein the step of finding a rotational shift includes steps of:

finding a first location (x₁, y₁) at which the inverse Fourier transform of the rotational shift is maximized; and

finding a second location (x₂, y₂) among four points (x₁±

1, y₁±

1) at which the inverse Fourier transform of the rotational shift is the largest.

17. The machine readable storage medium of claim 12, wherein the step of finding the translational shift comprises steps of:

finding a first location (x₁, y₁) at which the inverse Fourier transform of the translational shift is maximized;

finding a second location (x₂, y₂) among four points (x₁±

1, y₁±

1) at which the inverse Fourier transform of the translational shift is the largest; and

finding the translational shift from formulas

$x_T=\frac{w_{x<br><br><br><br>1}<br><br>x_1+w_{x<br><br><br><br>2}<br><br>x_2}{w_{x<br><br><br><br>1}+w_{x<br><br><br><br>2}}$ $y_T=\frac{w_{y<br><br><br><br>1}<br><br>y_1+w_{y<br><br><br><br>2}<br><br>y_2}{w_{y<br><br><br><br>1}+w_{y<br><br><br><br>2}}$

where (x_T, y_T) is the translational shift, and w_xi and w_yi are defined by formulas

w_xi=∫

(|F(x_i,y₁)|)+∫

(|F(x_i,y₂)|)

w_yi=∫

(|F(x₁,y_i)|)+∫

(|F(x₂,y_i)|)

where i stands for 1 or 2, F stands for Fourier transform, and ∫

is an empirical function.

18. The machine readable storage medium of claim 17, wherein the empirical function includes a function of ∫

(z)=z^α

, where α

is a parameter.

19. The machine readable storage medium of claim 12, the operations further comprising:

applying an erosion-dilation filter to process a difference image of the reference image and the transformed first-transformed sample image; and

reconstructing the transformed first-transformed sample image by adding the difference image to the reference image.