SAR and FLIR image registration method

• US 6,795,590 B1
• Filed: 09/22/2000
• Issued: 09/21/2004
• Est. Priority Date: 09/22/2000
• Status: Expired due to Fees

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Chen, Yang
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US09/668,657
• Time in Patent Office: 1,460 Days
• Field of Search: 382/294, 382/151, 382/154, 382/103, 382/280, 348/135, 348/139, 342/93, 342/451, 342/189, 342/191
• US Class Current: 382/294
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/37 using transform domain methods
• CPC Class Codes: G06V 10/753 Transform-based matching, e.g. Hough transform

Abstract

There is disclosed herein, a method used to associate or correspond synthetic aperture radar (SAR) and forward looking infrared (FLIR) images. Based on feature points detectable from both images, a two stage approach is taken to address the issue of SAR and FLIR image registration: an initial registration stage where feature points detected from the FLIR image are transformed into the SAR image coordinates; and a residual registration stage where the SAR and FLIR feature points undergo a “Generalized Hough Transform” from which a maximal subset of matching feature points is obtained and the registration transformation can be derived. These two stages are separated into five steps which comprise the SAR and FLIR Image Registration Method: (1) extracting feature points from said SAR and FLIR images; (2) creating an initial registration of the FLIR image; (3) creating a two-dimensional residual registration utilizing a generalized Hough transform; (4) estimating the registration transformation; and, (5) verifying the registration. This method allows the residual registration to be done in a two-dimensional (rather than six) Hough transform, which results in fast and robust implementation as well as reduce the possibility of false registration.

Claims

1. A method for obtaining a set of matching feature points from SAR and FLIR images, comprising the following steps:

extracting feature points from SAR and FLIR images;

creating an initial registration of said FLIR image with respect to the SAR image;

determining a set of matching feature points by utilizing a generalized Hough transform on the extracted feature points from the SAR and FLIR images;

estimating a registration transformation based on the set of matching feature points; and

then,

verifying the registration using the estimated registration transformation.

2. A method for obtaining a set of matching feature points from SAR and FLIR images as recited in claim 1, wherein the extracting step comprises:

using an algorithm such as a constant false-alarm rate (CFAR) detector.

3. A method for matching feature points from said SAR and FLIR images as recited in claim 1, wherein the SAR image has a ground plane and X, Y, and Z axes, wherein the FLIR image has a ground plane and X, Y, and Z axes, and wherein the step of creating an initial registration comprises:

using a homogeneous coordinate system;

$\begin{array}{cc}\left[\begin{array}{c}u<br><br>\text{\hspace{1em}<br><br>}<br><br>X\\ u<br><br>\text{\hspace{1em}<br><br>}<br><br>Y\\ u\end{array}\right]=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& -1/f\end{array}\right]<br><br>P& \mathrm{Eq}.\text{\hspace{1em}<br><br>}<br><br>\left(1\right)\end{array}$

where P is a three-dimensional (3-D) point in a camera coordinate frame having an origin and representing a point being imaged, (X, Y) is the image of P, u is a scaling factor, and f is a camera focal length;

representing P by its coordinates (x, y, z)^t in a reference frame, W, having a XOY plane, as follows;

$\begin{array}{cc}P=R_r<br><br>R_d<br><br>R_a<br><br>\left(\left[\begin{array}{c}x\\ y\\ z\end{array}\right]-\left[\begin{array}{c}{t}_x\\ t_y\\ t_z\end{array}\right]\right)& \mathrm{Eq}.\text{\hspace{1em}<br><br>}<br><br>\left(2\right)\end{array}$

where R_r, R_d and R_a are 3 by 3 orthonormal rotation matrices representing a roll angle θ

_r, a depression angle θ

_d, and an azimuth angle θ

_a of the camera coordinate frame relative to the reference frame, and (t_x, t_y, t_z)^t is a position vector representing the origin of the camera coordinate frame in the reference frame W;

assuming that the reference frame is such that its XOY plane coincides with the SAR image ground plane, its X and Y axes coincide with X and Y axes of the SAR image, respectively, and its Z axis points upwards relative to the image ground plane, and that all points in the FLIR images lie on a ground plane with z=h representing a ground plane elevation relative to the reference frame W;

solving for x and y, resulting in a back-projection transformation from an image point (X, Y) to a 3-D point (x, y, z);

$\begin{array}{cc}x=\frac{A_1<br><br>X+A_2<br><br>Y+A_3}{A_7<br><br>X+A_8<br><br>Y+A_9},y=\frac{A_4<br><br>X+A_5<br><br>Y+A_6}{A_7<br><br>X+A_8<br><br>Y+A_9},z=h& \mathrm{Eq}.\text{\hspace{1em}<br><br>}<br><br>\left(3\right)\end{array}$

where

A₁=cos(θ

_d)sin(θ

_r)t_x+(h−

t_z)(sin(θ

_a)sin(θ

_d)sin(θ

_r)−

cos(θ

_a)cos(θ

_r))

A₂=−

cos(θ

_d)cos(θ

_r)t_x−

(h−

t_z)(sin(θ

_a)sin(θ

_d)cos(θ

_r)+cos(θ

_a)sin(θ

_r))

A₃=f(sin(θ

_d)t_x−

sin(θ

_a)cos(θ

_d)(h−

t_z))

A₄=cos(θ

_d)sin(θ

_r)t_y+(h−

t_z)(sin(θ

_a)cos(θ

_r)+cos(θ

_a)sin(θ

_d)sin(θ

_r))

A₅=cos(θ

_d)cos(θ

_r)t_y+(h−

t_z)(sin(θ

_a)sin(θ

_r)−

cos(θ

_a)sin(θ

_d)cos(θ

_r))

A₆=f(sin(θ

_d)t_y−

cos(θ

_a)cos(θ

_d)(h−

t_z))

A₇=cos(θ

_d)sin(θ

_r)

A₈=−

cos(θ

_d)cos(θ

_r)

A₉=f sin(θ

_d)

applying the back-projection transformation of Equation (3) for each feature point in the FLIR image;

scaling the back-projected feature points into the SAR image coordinates as follows;

$\begin{array}{cc}\left[\begin{array}{c}x<br><br>\text{\hspace{1em}<br><br>}<br><br>s\\ y<br><br>\text{\hspace{1em}<br><br>}<br><br>s\end{array}\right]=c<br><br>\left[\begin{array}{c}x\\ y\end{array}\right],& \mathrm{Eq}.\text{\hspace{1em}<br><br>}<br><br>\left(4\right)\end{array}$

where (x, y)^t is the back-projection of the FLIR feature point in the reference frame W, (xs, ys) is the back-projection in the SAR image, and c is a constant reflecting the resolution of the SAR image; and

then,

generating {(xs_i, ys_t)}, i=1, . . . , N as a set of back-projected FLIR feature points, and {(xr_i, yr_t)}, i=1, . . . , M as a set of SAR feature points.

4. A method for matching feature points from SAR and FLIR images as recited in claim 3, wherein the step of determining a set of matching feature points is performed by:

determining an array size from the span of the back-projected FLIR feature points and the SAR feature points;

having a Generalized Hough Transform array of size (X_max−

X_min)/q+1 by (Y_max−

Y_min)/q+1, where q represents a quantization unit;

determining the quantization unit q according to a window size w;

┌

w/4┐

, where ┌

x┐

is the smallest integer greater than x; and

then,

using the five-step Generalized Hough Transform (GHT) to determine the set of matching feature points.

5. A method for matching feature points from SAR and FLIR images as set forth in claim 1, wherein the estimation step comprises:

using a Least-Squares Method or, alternatively, an Inverse Jacobian Method; and

then,

updating said registration transformation using an Inverse Jacobian Method.

6. A method for matching feature points from SAR and FLIR images as set forth in claim 1, wherein the SAR images each comprise a plurality of feature points, with each feature point comprised of pixels, with each pixel represented by x and y coordinates, wherein the FLIR images each comprise a plurality of feature points, with each feature point comprised of pixels, with each pixel represented by x and y coordinates, and wherein the verification step comprises:

transforming all of the FLIR feature points into SAR coordinates using a Least-Squares Method or, alternatively, an Inverse Jacobian Method to generate a set of newly estimated parameters B₁ to B₈;

performing a search for the closest SAR feature point to each transformed FLIR feature point within a distance of 4 pixels in both x and y coordinates;

adding the pairs of FLIR and SAR feature points to the two-dimensional residual registration if such SAR feature points are found; and

then, checking the two-dimensional residual registration, and determining the registration to be complete when it contains more than 4 points.