Mechanism for registering digital images obtained from multiple sensors having diverse image collection geometries

• US 5,550,937 A
• Filed: 09/12/1994
• Issued: 08/27/1996
• Est. Priority Date: 11/23/1992
• Status: Expired due to Term

• Current Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Original Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Inventors: Bryan, Joseph K., Black, Stephen B., Bell, David M.
• Primary Examiner(s): Mancuso, Joseph
• Assistant Examiner(s): Stokey, Richard
• Application Number:
  US08/304,316
• Time in Patent Office: 715 Days
• Field of Search: 382/100, 382/107, 382/113, 382/154, 382/191, 382/276, 382/278, 382/293, 382/294, 382/295, 382/296, 382/302, 382/130-132, 356/390, 356/399, 356/400, 356/401, 358/450, 364/731, 342/25, 348/699, 348/701, 395/127, 395/135
• US Class Current: 382/293
• CPC Class Codes: G06T 3/153 using elastic snapping
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: G06T 7/37 using transform domain methods

Abstract

First and second digital images, such as terrestrial images captured by one or more spaceborne or airborne sensors, are transformed onto a registration surface in accordance with geometry models of the sensors, to bring the digital images into mutual alignment on the registration surface. The images are initially subsampled to obtain reduced spatial resolution versions of each image. For a base resolution version of each image, respective neighborhoods of pixels associated with matching point grids in each image are defined. Edge-preemphasized contents of respective neighborhoods of pixels of each image are cross-correlated with one another, to derive a measure of misregistration between the images. One or both of the geometry models of the sensors are modified in accordance with the measure of misregistration, so as to effectively decrease misregistration between the images on the registration surface. The neighborhood cross-correlation process is then repeated for increasingly higher spatial resolution versions of each image, to effect a coarse-to-fine improvement in registration error correction.

Claims

1. For use with a digital image processing system in which plural images of an object are derived by a plurality of sensors, object observation and image collection parameters of which may be different from one another, and wherein said plural images of said object obtained by said plurality of sensors are digitized to form first and second digital images, said first digital image having a first array of pixels of a prescribed spatial resolution, and said second digital image having a second array of pixels of said prescribed spatial resolution, a method of bringing said first and second digital images into effective registration with one another comprising the steps of:

(a) processing said first and second digital images to obtain plural versions of each of said first and second digital images, having successively reduced spatial resolutions from said prescribed spatial resolution to a prescribed base resolution;

(b) comparing image contents of respective first neighborhoods of pixels associated with respective points of a first array of points within a first selected resolution version of said first digital image with image contents of respective second neighborhoods of pixels associated with respective points of a second array of points within said first selected resolution version of said second digital image, so as to derive a measure of misregistration between said first and second images, the contents of each of said first and second neighborhoods of pixels being defined with respect to a reference image surface in accordance with first and second geometric models of sensors through which said first and second digital images are obtained, said measure of misregistration between said first and second digital images being derived by

(b1) comparing the extent to which a prescribed content of respective first neighborhoods of pixels associated with said first array of points within said first selected resolution version of said first digital image matches a prescribed content of respective second neighborhoods of pixels associated with said second array of points within said first selected resolution version of said second digital image, by

(b1-1) emphasizing said prescribed content of said first and second neighborhoods of pixels, and

(b1-2) comparing respective prescribed content-emphasized first neighborhoods of pixels with respective edge content-emphasized second neighborhoods of pixels, so as to derive a distribution of peaks representative of said measure of misregistration between said first and second digital images, such that said measure of misregistration between said first and second digital images is in accordance with offsets of peaks from points of said first and second arrays of points respectively,

(b3) generating respective consistent subsets of offsets of peaks, which subsets consistently point in the same direction from points of said first and second arrays of points, respectively, said peaks falling within a prescribed range of offset values,

(b4) processing said consistent subsets of offsets to derive, for each consistent subset, a measure of the accuracy to which that subset is representative of misregistration between said first and second images,

(b5) comparing respective measures of accuracy derived for respective consistent subsets, and

(b6) deriving said measure of misregistration between said first and second digital images, in accordance with said comparing step (b5) of respective measures of accuracy derived for respective consistent subsets; and

(c) modifying at least one of said first and second geometric models of the sensors, through which said first and second digital images are obtained, in accordance with the measure of misregistration derived in step (b6), so as to effectively decrease misregistration between said first and second images.

2. A method according to claim 1, wherein said step (b1-2) of comparing comprises correlating respective prescribed content-emphasized first neighborhoods of pixels with respective edge content-emphasized second neighborhoods of pixels, so as to derive a distribution of correlation peaks representative of said measure of misregistration between said first and second digital images, such that said measure of misregistration between said first and second digital images is in accordance with offsets of correlation peaks from points of said first and second arrays of points, respectively, and generating respective consistent subsets of offsets of correlation peaks, which subsets consistently point in the same direction from points of said first and second arrays of points respectively, said correlation peaks falling within a prescribed range of offset values.

3. A method according to claim 2, wherein step (b) further comprises generating respective subsets of measurements representative of offsets of correlation peaks from points of said first and second arrays of points, and shapes of said correlation peaks that fall within a prescribed range of measurement values, and processing said subsets of measurements to derive, for each subset, a quality metric representative of the accuracy to which that subset is representative of misregistration between said first and second images, and deriving said measure of misregistration between said first and second images in accordance with a comparison of respective quality metrics derived for respective ones of said subsets.

4. A method according to claim 3, wherein step (b) further comprises comparing the difference between the two largest quality metrics with a prescribed threshold value and, in response to said difference exceeding said prescribed threshold value, deriving said measure of misregistration between said first and second digital images in accordance with the subset of measurement having the largest quality metric.

5. A method according to claim 3, wherein step (b) further comprises comparing the difference between the two largest quality metrics with a prescribed threshold value and, in response to said difference failing to exceed said prescribed threshold value, repeating step (b) using first and second arrays of points having an increased spatial resolution.

6. A method according to claim 5, wherein step (b) further comprises comparing the difference between the two largest quality metrics with a prescribed threshold value and, in response to said difference exceeding said prescribed threshold value, deriving said measure of misregistration between said first and second digital images in accordance with the subset of measurement having the largest quality metric, but, in response to said difference failing to exceed said prescribed threshold value, repeating step (b) using first and second arrays of points having a further increased spatial resolution.

7. A method according to claim 6, wherein, in step (b), in response to said difference failing to exceed said prescribed threshold value after a predetermined number of repetitions of step (b), using first and second arrays of points having successively increased values of spatial resolution, deriving said measure of misregistration between said first and second images in accordance with the subset of measurement having the largest quality metric.

8. A method according to claim 1, wherein step (c) further comprises modifying said at least one of said first and second geometric models of the sensors in such a manner as to effect a minimal perturbation of parameters of said first and second geometric models of said sensors, and performing conjugate gradient processing of a set of n space parameters through which geometric models of said sensors are defined.

9. A method according to claim 2, wherein, in step (b1-2), said step of comparing comprises correlating said respective edge content-emphasized first neighborhoods of pixels with respective edge content-emphasized second neighborhoods of pixels by Fast Fourier Transform processing.

10. A method according to claim 2, wherein, in step (b1-2), said step of comparing comprises correlating said respective edge content-emphasized first neighborhoods of pixels with respective edge content-emphasized second neighborhoods of pixels by a parametric hill finding routine.