Method and apparatus for performing geo-spatial registration using a Euclidean representation

• US 6,587,601 B1
• Filed: 06/28/2000
• Issued: 07/01/2003
• Est. Priority Date: 06/29/1999
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Hsu, Stephen Charles, Samarasekera, Supun
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US09/605,915
• Time in Patent Office: 1,098 Days
• Field of Search: 382/294, 382/295, 382/296, 382/154, 382/305, 382/312, 382/254, 382/276, 382/289, 382/284, 382/293, 702/5, 707/1, 707/100, 356/138, 356/139.09, 345/648, 345/419, 345/427, 345/651, 701/215, 348/135, 348/144
• US Class Current: 382/294
• CPC Class Codes: G01C 11/00 Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/33 using feature-based methods

Abstract

A system and method for accurately mapping between camera coordinates and geo-coordinates, called geo-spatial registration, using a Euclidean model. The system utilizes the imagery and terrain information contained in the geo-spatial database to precisely align geographically calibrated reference imagery with an input image, e.g., dynamically generated video images, and thus achieve a high accuracy identification of locations within the scene. When a sensor, such as a video camera, images a scene contained in the geo-spatial database, the system recalls a reference image pertaining to the imaged scene. This reference image is aligned very accurately with the sensor'"'"'s images using a parametric transformation produced by a Euclidean model. Thereafter, other information that is associated with the reference image can easily be overlaid upon or otherwise associated with the sensor imagery.

Claims

1. A system for performing geo-spatial registration of an input image and geographically calibrated reference imagery and a digital elevation map (DEM) comprising:

a reference database module containing geographically calibrated reference imagery and DEM, for producing geographically calibrated reference imagery and DEM relating to imagery in said input image; and

an alignment module, coupled to said reference database module, for aligning said input image to said geographically calibrated reference imagery and DEM using a Euclidean model.

2. The system of claim 1 further comprising:

a mosaic generation module for producing an image mosaic as said input image, where said image mosaic is generated from a sequence of sensor images.

3. The system of claim 1 further comprising:

a source of sensor attitude for generating attitude information pertaining to a sensor producing said input image.

4. The system of claim 3 wherein said alignment module further comprises:

a coarse alignment block, coupled to said reference database module and said sensor attitude source, for aligning said geographically calibrated reference imagery and DEM to alignment with said input image using said sensor attitude.

5. The system of claim 1 wherein said alignment module further comprises:

a coarse alignment block, coupled to said reference database module, for aligning said geographically calibrated reference imagery and DEM.

6. The system of claim 4 wherein said alignment module further comprises:

a fine alignment block, coupled to said coarse alignment block, for accurately aligning said input image to said geographically calibrated reference imagery and DEM to a sub-pixel accuracy.

7. The system of claim 3 wherein said alignment module further comprises a synthetic view generation block for warping said geographically calibrated reference imagery and DEM to have a viewpoint similar to a viewpoint of said sensor.

8. The system of claim 3 wherein said sensor is a video camera.

9. The system of claim 5 further comprising a tracking block, coupled to said fine alignment block, for using a previously computed parametric transformation associated with a prior frame as an initialization transformation for computing a parametric transformation for a present frame.

10. The system of claim 5 further comprising a tracking block, coupled to said fine alignment block, for tracking said parametric transformation such that a new parametric transformation does not have to be computed for each new input image.

11. The system of claim 1 further comprising a display module for generating a display that uses the parametric transformation to align said geographically calibrated reference imagery with said input image and simultaneously display said geographically calibrated reference imagery and said input image.

12. The system of claim 4 wherein said coarse alignment block further comprises:

a input image salient feature extractor;

a reference image salient feature extractor;

an exhaustive search engine; and

a directed matching processor.

13. The system of claim 5 wherein said fine alignment block aligns imagery using the Euclidean model:

14. A method for performing geo-spatial registration of an input image and geographically calibrated reference imagery and digital elevation map (DEM) comprising:

producing geographically calibrated reference imagery and DEM relating to imagery in said input image; and

aligning said input image to said geographically calibrated reference imagery and DEM using a Euclidean model.

15. The method of claim 14 further comprising the step of:

producing an image mosaic as said input image, where said image mosaic is generated from a sequence of sensor images.

16. The method of claim 14 further comprising the step of:

generating attitude information pertaining to a sensor producing said input image.

17. The method of claim 16 wherein said aligning step further comprises the step of:

coarsely aligning said geographically calibrated reference imagery and DEM to alignment with said input image using said sensor attitude.

18. The method of claim 17 wherein said aligning step further comprises the step of:

accurately aligning said input image to said geographically calibrated reference imagery and DEM to a sub-pixel accuracy.

19. The method of claim 17 wherein said aligning step further comprises warping said geographically calibrated reference imagery and DEM to have a viewpoint similar to a viewpoint of said sensor.

20. The method of claim 18 further comprising the step of:

tracking said parametric transformation such that a new parametric transformation does not have to be computed for each new input image.

21. The method of claim 14 further comprising the step of:

generating a display that uses the parametric transformation to align said geographically calibrated reference imagery with said input image and simultaneously display said geographically calibrated reference imagery and said input image.

22. The method of claim 18 wherein said accurate aligning step further comprises the step of:

selecting an alignment process best suited for the input image.

23. The method of claim 14 further comprising the step of:

determining geographic coordinates of a user selected point within said input image.

24. The method of claim 14 further comprising the step of:

compensating for lens distortion associated with the input image.

25. The method of claim 14 further comprising the step of:

updating the geographically calibrated reference imagery in the reference database with information from the input image.

26. The method of claim 14 further comprising the step of:

generating a synthetic fly through starting at a current viewpoint of the input image and continuing using the geographically calibrated reference imagery.

27. The method of claim 17 wherein said coarsely aligning step further comprises:

extracting salient features from the input image;

extracting salient features from the a reference image;

exhaustively searching the salient features of the input image and the reference image; and

identifying matching salient features in the input image and the reference image.

28. The method of claim 18 wherein said accurately aligning aligns imagery using the Euclidean model:

29. A digital storage medium containing a computer program that, when executed by a general purpose computer, forms a specific purpose computer that performs the steps of:

producing geographically calibrated reference imagery and digital elevation map (DEM) relating to imagery in said input image; and

aligning said input image to said geographically calibrated reference imagery and DEM using a Euclidean model.

30. The medium of claim 29 further performing the step of:

producing an image mosaic as said input image, where said image mosaic is generated from a sequence of sensor images.