System and method for automatically registering an image to a three-dimensional point set

• US 9,269,145 B2
• Filed: 10/22/2013
• Issued: 02/23/2016
• Est. Priority Date: 10/23/2012
• Status: Active Grant

• Current Assignee: Raytheon Company (Rtx Corporation)
• Original Assignee: Raytheon Company (Rtx Corporation)
• Inventors: Ely, Richard W., Verret, Jody D., Boroughs, Grant B.
• Primary Examiner(s): Nguyen, Phu K
• Application Number:
  US14/059,802
• Publication Number:
  US 20140112536A1
• Time in Patent Office: 854 Days
• Field of Search: 345/419, 345/428, 345/426, 345/631, 345/632, 382/103
• US Class Current: 1/1
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 15/60 Shadow generation
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30184 Infrastructure
• CPC Class Codes: G06T 2207/30212 Military
• CPC Class Codes: G06T 7/32 using correlation-based methods

Abstract

A system and methods can create a synthetic image of a target from a 3D data set, by using an electro-optical (EO) image and sun geometry associated with the EO image. In some examples, a 3D surface model is created from a 3D data set. The 3D surface model establishes a local surface orientation at each point in the 3D data set. A surface shaded relief (SSR) is produced from the local surface orientation, from an EO image, and from sun geometry associated with the EO image. Points in the SSR that are in shadows are shaded appropriately. The SSR is projected into the image plane of the EO image. Edge-based registration extracts tie points from the projected SSR. The 3D data set converts the tie points to ground control points. A geometric bundle adjustment aligns the EO image geometry to the 3D data set.

Claims

1. A system for producing a synthetic image from three-dimensional data set that includes a plurality of three-dimensional data points, the system comprising processing circuitry arranged to:

assign to each three-dimensional data point an intensity that is proportional to a dot product of a sun vector angle and a surface local unit normal vector at the three-dimensional data point;

cast shadows and modify the assigned intensity values via an atmospheric scattering model;

perform a correlation process at a grid of points between projected surface shaded relief and electro-optical images to generate a grid of correlated point locations;

compare correlation quality metrics for each correlated point location in the grid to specified thresholds; and

retain the correlated point locations in the grid that exceed the specified thresholds.

2. The system of claim 1, wherein assigning to each three-dimensional data point the intensity that is proportional to the dot product of the sun vector angle and the surface local unit normal vector at the three-dimensional data point comprises:

computing the sun'"'"'s local azimuth and elevation angles from an acquisition date and time of an electro-optical image.

3. The system of claim 1, wherein assigning to each three-dimensional data point the intensity that is proportional to the dot product of the sun vector angle and the surface local unit normal vector at the three-dimensional data point comprises:

extracting the sun vector angle from metadata of an electro-optical image.

4. The system of claim 1, wherein modifying the assigned intensity values via the atmospheric scattering model comprises:

producing a surface shaded relief value for each three-dimensional data point, using the three-dimensional data points as elevation data and casting shadows.

5. The system of claim 1, wherein performing the correlation process at the grid of points between projected surface shaded relief and electro-optical images comprises:

rasterizing the surface shaded relief values;

projecting the rasterized surface shaded relief values to the space of the electro-optical image using associated XYZ locations provided by the three-dimensional data set to form a projected surface shaded relief image; and

performing image correlations between the projected surface shaded relief image and the electro-optical image.

6. The system of claim 5, wherein performing the correlation process at the grid of points between projected surface shaded relief and electro-optical images further comprises:

obtaining X,Y,Z Universal Transverse Mercator (UTM) coordinates for respective projected surface shaded relief image coordinates;

converting the X,Y,Z UTM coordinates to geodetic coordinates; and

producing ground control points (GCP) and corresponding electro-optical image coordinates from the geodetic coordinates.

7. The system of claim 6, wherein the geodetic coordinates include latitude, longitude, and height.

8. The system of claim 6, wherein each ground control point includes values of latitude, longitude, elevation, and corresponding EO imageSample and imageLine coordinates.

9. The system of claim 8, wherein the processing circuitry is further arranged to:

use the latitude, longitude, and height and corresponding image coordinates as ground control points in a photogrammetric bundle adjustment of the electro-optical image.

10. A system for producing a synthetic image from three-dimensional data set that includes a plurality of three-dimensional data points, the system comprising processing circuitry arranged to:

input a three-dimensional data set that includes a plurality of three-dimensional data points;

generate a rasterized elevation map from the three-dimensional data points;

generate a rasterized surface shaded relief from the rasterized elevation map, for a sun elevation and azimuth at a time of collection for an electro-optical image;

project the rasterized surface shaded relief into a space of the electro-optical image;

tile the rasterized surface shaded relief into a grid;

register each tile independently;

retain tiles having correlating quality metrics that exceed respective specified thresholds;

for each retained tile, extract a ground control point; and

perform a geometric bundle adjustment of a geometry of the electro-optical image using the extracted ground control points.

11. The system of claim 10, further comprising, between inputting the three-dimensional data set that includes the plurality of three-dimensional data points and generating the rasterized elevation map from the three-dimensional data points:

removing outliers from the three-dimensional data points.

12. The system of claim 10, further comprising, between generating the rasterized elevation map from the three-dimensional data points and generating the rasterized surface shaded relief from the rasterized elevation map:

filling in voids in the rasterized elevation map using interpolation and hole filling.

13. The system of claim 10, wherein the rasterized surface shaded relief is projected into the space of the electro-optical image using the three-dimensional data points and rational polynomial coefficients from the electro-optical image.

14. The system of claim 10, wherein each tile is registered independently by correlating edges extracted from the projected surface shaded relief with the electro-optical image.

15. The system of claim 10, wherein, for each retained tile, the ground control point is extracted using a three-dimensional data point closest to a center of the tile and a corresponding electro-optical image location derived from the edge correlation.

16. The system of claim 10, wherein generating a rasterized surface shaded relief from the rasterized elevation map comprises:

generating a rasterized surface shaded relief by computing an angle between a local surface normal and a sun ray;

marking three-dimensional data points in the rasterized elevation map that are in shadow;

modifying shadow point grayshades using an atmospheric scattering model; and

updating the rasterized surface shaded relief with the modified shadow point grayshades.

17. A system for producing a synthetic image from three-dimensional data set that includes a plurality of three-dimensional data points, the system comprising processing circuitry arranged to:

input a three-dimensional data set that includes a plurality of three-dimensional data points;

remove outliers from the three-dimensional data points;

generate a rasterized elevation map from the three-dimensional data points;

fill in voids in the rasterized elevation map using interpolation and hole filling;

generate a rasterized surface shaded relief from the rasterized elevation map, for a sun elevation and azimuth at a time of collection for an electro-optical image;

project the rasterized surface shaded relief into a space of the electro-optical image using the three-dimensional data points and rational polynomial coefficients from the electro-optical image;

tile the rasterized surface shaded relief into a grid;

register each tile independently by correlating edges extracted from the projected surface shaded relief with the electro-optical image;

retain tiles having correlating quality metrics that exceed respective specified thresholds;

for each retained tile, extract a ground control point using a three-dimensional data point closest to a center of the tile and a corresponding electro-optical image location derived from the edge correlation; and

perform a geometric bundle adjustment of a geometry of the electro-optical image using the extracted ground control points.

18. The system of claim 17, wherein generating the rasterized surface shaded relief from the rasterized elevation map comprises:

generating a rasterized surface shaded relief by computing an angle between a local surface normal and a sun ray;

marking three-dimensional data points in the rasterized elevation map that are in shadow;

modifying shadow point grayshades using an atmospheric scattering model; and

updating the rasterized surface shaded relief with the modified shadow point grayshades.