Method and apparatus for performing geo-spatial registration

• US 6,512,857 B1
• Filed: 05/08/1998
• Issued: 01/28/2003
• Est. Priority Date: 05/09/1997
• Status: Expired due to Term

• Current Assignee: SRI International, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Sawhney, Harpreet Singh, Hsu, Stephen Charles, Asmuth, Jane, Kumar, Rakesh
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): Patel, Kanji
• Application Number:
  US09/075,462
• Time in Patent Office: 1,726 Days
• Field of Search: 382/284, 382/285, 382/287, 382/289, 382/294, 382/295, 382/296, 382/190, 382/305, 382/115, 382/103, 382/106, 702/5, 702/85, 345/121, 345/126, 345/435-438, 244/317, 348/117, 701/28, 701/207, 701/208, 701/213, 701/25, 701/209, 701/211
• US Class Current: 382/294
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

A system and method for accurately mapping between camera coordinates and geo-coordinates, called geo-spatial registration. The system utilizes the imagery and terrain information contained in the geo-spatial database to precisely align geographically calibrated reference imagery with an input image, e.g., dynamically generated video images, and thus achieve a high accuracy identification of locations within the scene. When a sensor, such as a video camera, images a scene contained in the geo-spatial database, the system recalls a reference image pertaining to the imaged scene. This reference image is aligned very accurately with the sensor'"'"'s images using a parametric transformation. Thereafter, other information that is associated with the reference image can easily be overlaid upon or otherwise associated with the sensor imagery.

Claims

1. A system for performing geo-spatial registration of an input image and geographically calibrated reference imagery comprising:

a reference database module containing geographically calibrated reference imagery, for producing geographically calibrated reference imagery relating to imagery in said input image; and

an alignment module, coupled to said reference database module, for aligning said input image to said geographically calibrated reference imagery, where said alignment module comprises;

a coarse alignment block, coupled to said reference database module and a sensor attitude source, for aligning said geographically calibrated reference imagery to alignment with said input image by using attitude information generated from said sensor attitude source, where said attitude information pertains to a sensor producing said input image, and

a fine alignment block, coupled to said coarse alignment block, for accurately aligning said input image to said geographically calibrated reference imagery to a sub-pixel accuracy.

2. The system of claim 1 further comprising:

a mosaic generation module for producing an image mosaic as said input image, where said image mosaic is generated from a sequence of sensor images.

3. The system of claim 1 wherein said alignment module further comprises a synthetic view generation block for warping said geographically calibrated reference imagery to have a viewpoint similar to a viewpoint of said sensor.

4. The system of claim 1 wherein said sensor is a video camera.

5. The system of claim 1 further comprising a tracking block, coupled to said fine alignment block, for tracking said parametric transformation such that a new parametric transformation does not have to be computed for each new input image.

6. The system of claim 1 further comprising a display module for generating a display that uses a parametric transformation to align said geographically calibrated reference imagery with said input image and simultaneously display said geographically calibrated reference imagery and said input image.

7. The system of claim 1 wherein said coarse alignment block further comprises:

a input image salient feature extractor;

a reference image salient feature extractor;

an exhaustive search engine; and

a directed matching processor.

8. The system of claim 1 wherein said fine alignment block further comprises:

a fine alignment process selector that selects an alignment process best suited for the input image.

9. The system of claim 8 wherein the fine alignment selector can select an alignment process from the following:

a projective pose process, a projective process, or a plane+parallax process.

10. A method for performing geo-spatial registration of an input image and geographically calibrated reference imagery comprising:

producing geographically calibrated reference imagery relating to imagery in said input image; and

aligning said input image to said geographically calibrated reference imagery, where said aligning comprises

coarsely aligning said geographically calibrated reference imagery to alignment with said input image by using attitude information generated from a sensor attitude source, where said attitude information pertains to a sensor producing said input image, and

accurately aligning said input image to said geographically calibrated reference imager to a sub-pixel accuracy.

11. The method of claim 10 further comprising the step of:

producing an image mosaic as said input image, where said image mosaic is generated from a sequence of sensor images.

12. The method of claim 10 wherein said aligning step further comprises warping said geographically calibrated reference imagery to have a viewpoint similar to a viewpoint of said sensor.

13. The method of claim 10 further comprising the step of:

tracking a parametric transformation such that a new parametric transformation does not have to be computed for each new input image.

14. The method of claim 10 further comprising the step of:

generating a display that uses a parametric transformation to align said geographically calibrated reference imagery with said input image and simultaneously display said geographically calibrated reference imagery and said input image.

15. The method of claim 10 wherein said accurate aligning step further comprises the step of:

selecting an alignment process best suited for the input image.

16. The method of claim 10 further comprising the step of:

determining geographic coordinates of a user selected point within said input image.

17. The method of claim 10 further comprising the step of:

displaying the input image warped and superimposed on the geographically calibrated reference imagery.

18. The method of claim 10 further comprising the step of:

updating the geographically calibrated reference imagery in a reference database with information from the input image.

19. The method of claim 10 further comprising the step of:

generating a synthetic fly through starting at a current viewpoint of the input image and continuing using the geographically calibrated reference imagery.

20. A digital storage medium containing a computer program that, when executed by a general purpose computer, forms a specific purpose computer that performs the steps of:

producing geographically calibrated reference imagery relating to imagery in an input image; and

aligning said input image to said geographically calibrated reference imagery, where said aligning comprises

coarsely aligning said geographically calibrated reference imagery to alignment with said input image by using attitude information generated from a sensor attitude source, where said attitude information pertains to a sensor producing said input image, and

accurately aligning said input image to said geographically calibrated reference imagery to a sub-pixel accuracy.

21. The medium of claim 20 further performing the step of:

producing an image mosaic as said input image, where said image mosaic is generated from a sequence of sensor images.