Method for autonomous image registration

• US 5,581,638 A
• Filed: 02/28/1995
• Issued: 12/03/1996
• Est. Priority Date: 07/26/1993
• Status: Expired due to Term

• Current Assignee: Raytheon Company (Rtx Corporation)
• Original Assignee: E-Systems LLC
• Inventors: Skinner, Thomas J., Ingram, Paul M., Givens, Fenton L.
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): Bella, Matthew C.
• Application Number:
  US08/395,633
• Time in Patent Office: 644 Days
• Field of Search: 382/154, 382/278, 382/103, 382/151, 382/293, 382/294, 382/299, 348/51, 348/263
• US Class Current: 382/294
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

A method for automatically registering an image pair having a common coverage area, wherein a pair of image data sets, each representing an image of an image pair, are processed to generate initial dewarping coefficients for the image data sets. The translational offset between the images is estimated using a phase correlation process. Next, the common coverage area of the data image sets is determined by finding the vertices defining the limits of the common coverage area. Conjugate point pairs are extracted from the common coverage area by defining a series of block and subblock grids and defining the center point of the grid as a conjugate point location. The determined conjugate point sets are used to generate geometrically corrected registered dewarping coefficients and the image data sets are resampled using the registered dewarping coefficients to generate a registered image pair.

Claims

1. A method for registering multiple autonomously images of a target, the method comprising the steps of:

detecting and recording the target at different geolocations to generate multiple, digitized images of the target;

storing the digital images in a storage area;

generating image registration in a processing unit receiving the digital images from the storage area by the steps of;

(a) dewarping the multiple digitized images into approximate registration using image geolocation information;

(b) generating an identification of areas of common coverage between the approximately registered digitized images;

(c) generating greatly reduced resolution digital images of areas of common coverage between the approximately registered images;

(d) phase correlating the areas of common coverage between the reduced resolution images to identify initial conjugate points;

(e) refining the identified initial conjugate points at increased image resolution to identify final conjugate points in the common coverage area between the approximately registered images; and

(f) dewarping the multiple images of the target into registration using the identified final conjugate points; and

stereoscopically displaying the registered multiple images at a display terminal.

2. The method as in claim 1 wherein the geolocation information comprises position, altitude and velocity data for each image.

3. The method as in claim 1 wherein the step of dewarping the multiple images into approximate registration comprises the steps of:

scaling the multiple images; and

horizontally orienting the scaled multiple images.

4. The method as in claim 1 wherein the step of generating an identification of the areas of common coverage comprises the steps of:

generating reduced resolution images of the approximately registered images;

phase correlating the reduced resolution images to determine a translational offset between the approximately registered images;

aligning the approximately registered images in accordance with the determined translational offset;

boundary tracking along the aligned, approximately registered images; and

comparing the images during boundary tracking to locate vertices of the areas of common coverage.

5. The method as in claim 1 wherein the step of refining comprises the steps of:

generating an increase in the magnification of the areas of common coverage;

phase correlating the magnified areas of common coverage to locate refined conjugate points in the areas of common coverage between the approximately registered images;

repeating the step of generating an increase in the magnification and the step of phase correlating to establish a predetermined magnification of the areas of common coverage; and

outputting the located refined conjugate points as the final conjugate points.

6. The method as in claim 1 further including the steps of:

identifying registered conjugate points having a vertical parallax exceeding a predetermined minimum;

removing the conjugate points identified as exceeding the predetermined minimum; and

outputting remaining registered conjugate points as the final conjugate points.

7. A method for displaying at a display terminal first and second digital images of a target in registration, the images detected and recorded in a storage area, the images autonomously detected at different geolocations, comprising the steps of:

generating image registration in a processing unit receiving the digital images from the storage area by the steps of;

(a) converting the first and second digital images into the same scale in accordance with geolocation information for the images;

(b) generating initially dewarped first and second images by rotating the converted first and second images in accordance with the geolocation information to horizontally orient an epipolar line;

(c) comparing the initially dewarped first and second images to identify areas of common coverage between the first and second images;

(d) generating greatly reduced resolution images of the areas of common coverage between the initially dewarped first and second images;

(e) comparing the areas of common coverage between two reduced resolution images to identify initial conjugate points;

(f) generating final conjugate points in the common coverage area between the initially dewarped first and second images by refining the identified initial conjugate points at increased image resolution;

(g) converting the first and second images into the same scale in accordance with the registered final conjugate points; and

(h) generating finally dewarped first and second images by rotating the converted first and second images into registration in accordance with the registered final conjugate points to horizontally orient the epipolar line of each image; and

stereoscopically displaying the registered, finally dewarped first and second images at the terminal display.

8. The method as in claim 7 wherein the geolocation information comprises position, altitude and velocity data.

9. The method as in claim 7 wherein the step of comparing to identify the areas of common coverage comprises the steps of:

generating reduced resolution images of the initially dewarped images;

phase correlating the two reduced resolution images to determine a translational offset between the two approximately registered images;

aligning the two approximately registered images in accordance with the determined translational offset; and

locating vertices of the area of common coverage by comparing boundaries of the aligned images.

10. The method as in claim 7 wherein the step of generating final conjugate points comprises the steps of:

generating an increase in the magnification of the areas of common coverage;

comparing the magnified areas of common coverage to locate refined conjugate points in areas of common coverage between the two approximately registered images;

repeating the step of generating an increase in the magnification and the step of comparing to establish a predetermined magnification of the areas of common coverage; and

outputting the located refined conjugate points as the final conjugate points.

11. The method as in claim 10 wherein the step of outputting includes the steps of:

identifying refined conjugate points having a vertical parallax exceeding a predetermined minimum;

removing the refined conjugate points identified as exceeding the predetermined minimum; and

outputting remaining refined conjugate points as the final conjugate points.

12. A method for extracting conjugate points from between two autonomous sensor images, comprising the steps of:

(a) dewarping the two sensor images into approximate registration using sensor location information;

(b) generating an identification of areas of common coverage between the approximately registered images;

(c) generating greatly reduced resolution images of areas of common coverage between the approximately registered images;

(d) phase correlating the areas of common coverage between the reduced resolution images to identify initial conjugate points;

(e) adjusting the magnification of the phase correlated reduced resolution images;

(f) comparing the magnification adjusted images to refine the identification of the initial conjugate points;

(g) repeating steps (e) and (f) to establish a predetermined magnification; and

(h) outputting the refined initial conjugate points at the predetermined magnification as set of final conjugate points between the two sensor images.

13. The method of claim 12 wherein the step of phase correlating the reduced resolution images comprises the steps of:

overlaying each of the reduced resolution images with a block grid; and

phase correlating each block in one image with a corresponding block in the other image to identify the initial conjugate points in each correlated pair of blocks.

14. The method of claim 12 wherein the step of comparing the two magnification adjusted images comprises the steps of:

overlaying the two increased magnification images with a block grid;

overlaying the blocks within the block grid with a subblock grid; and

phase correlating each subblock in one image with a corresponding subblock in another image set to refine the identification of the initial conjugate points.

15. A method performed by a processing unit for registering images pairs having common scenery comprising:

storing in memory of a processing unit image data sets of first and second digital images of a common target taken at different geolocations;

initially sampling the image data sets of the first and second digital images stored in memory of the processing unit elements of at least one of the first and second images into a common coordinate system and scale according to initial dewarping parameters determined in response to sensor geolocation information for each image;

determining areas of common coverage between the initially sampled first and second images;

extracting from the first and second digital images pairs of conjugate points, one point in each pair from each of the first and second images, lying within the area of common coverage from the first and second images, the step of extracting including;

dividing the common coverage area of the initially sampled first and second images into a plurality of image blocks;

phase correlating corresponding image block pairs in the initially registered first and second images for determining average parallax over the entire block and predicting shift between blocks required to match the phases of the blocks; and

if a corresponding pair of image blocks phase correlate to within a predetermined degree of confidence, storing in memory of the processing unit the center of each block, shifted according to the predicted shift, as a conjugate pair; and

resampling a second time with the processing unit the first and second images according to refined dewarping parameters, the refined dewarping parameters being determined by geometric corrections of the initial dewarping parameters which result in at least a predetermined number of the conjugate pairs registering to within a predetermined minimum error after resampling.

16. The method of claim 15 wherein extracting conjugate pairs further comprises:

subdividing phase-correlated image block in each of the first and second images into image sub-blocks;

phase correlating corresponding image sub-blocks in each image and predicting a shift between image sub-blocks required to match the image sub-blocks; and

storing in memory the centers of correlated image sub-blocks, shifted according to the predicted shift, as conjugate pairs.

17. The method of claim 16 including reducing resolution of the initially registered first and second images before dividing the common coverage area into image blocks;

and further including increasing the resolution of the correlated image blocks before sub-dividing the image blocks.

18. An apparatus for automatically registering autonomous images pairs having common scenery comprising:

a memory for storing first and second digital images taken at different camera locations containing scenes of common terrain in memory;

means for determining, in response to receiving information describing sensor for each image, preliminary dewarping parameters;

means for initially sampling elements of at least one of the first and second images into a common coordinate system with the other of the first and second images according to the preliminary dewarping parameters;

means for determining areas of common coverage between the initially registered first and second images;

means for dividing the common coverage area of the initially registered first and second images into a plurality of image blocks;

means for phase correlating each image block in corresponding pairs of blocks by phase correlating them and predicting shift between image blocks required to match the image blocks;

means for storing, for each pair of image blocks which correlate, the center of each image block, shifted according to the predicted shift, as a conjugate pair;

means for geometrically correcting the initial dewarping parameters to produce refined dewarping parameters which result in at least a predetermined number of the conjugate pairs registering to within a predetermined minimum error; and

means for finally resampling the first and second images using the refined dewarping parameters.

19. The apparatus of claim 18 further comprising:

means for subdividing correlated image blocks into image sub-blocks;

means for phase correlating corresponding image sub-blocks and predicting a shift between image sub-blocks required to match the image blocks; and

means for storing in memory the centers of correlated image sub-blocks, shifted according to the predicted shift, as conjugate pairs.

20. The apparatus of claim 19 further comprising:

means for reducing resolution of the initially registered first and second images before dividing them into image blocks; and

means for increasing the resolution of the correlated image blocks before sub-dividing the image blocks.