Fusion of sensor data and synthetic data to form an integrated image

• US 20070297696A1
• Filed: 06/27/2006
• Published: 12/27/2007
• Est. Priority Date: 06/27/2006
• Status: Active Grant

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Hamza, Rida M., Feyereisen, Thea L.
• Granted Patent: US 7,925,117 B2
• Time in Patent Office: Days
• US Class Current: 382/294
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/35 using statistical methods

Abstract

A system and method for fusing sensor data and synthetic data to form an integrated image are described. An image registration process is used to fuse the images. The image registration process is performed on gradients of a landmark image. The gradients are extracted from both sensor and synthetic datasets for the landmark image. Using the image gradient, a center of mass for each of the gradient'"'"'s curves is calculated. By solving a minimization problem to reduce error, a desired transformation operator can be obtained to match the sensor image to the synthetic image.

Claims

1. A fusion method for combining sensor data with synthetic data to form an integrated image, comprising in combination:

identifying at least two landmarks;

extracting an image gradient from a sensor image dataset for each of the at least two landmarks;

extracting a corresponding image gradient from a synthetic image dataset for each of the at least two landmarks;

calculating a center of mass for each of the image gradients extracted from the sensor and synthetic image datasets;

calculating displacement between corresponding image gradients from the sensor and synthetic image datasets centered at the calculated centers of mass; and

stabilizing images by minimizing the displacement to form the integrated image.

2. The method of claim 1, wherein calculating displacement includes calculating a displacement vector that represents space shifting and rotation between the corresponding extracted image gradients.

3. The method of claim 2, wherein calculating displacement includes estimating the rotation using a rotation matrix.

4. The method of claim 2, wherein calculating displacement includes estimating the rotation as an average of two rotation angles.

5. The method of claim 1, wherein calculating displacement includes estimating an orientation of an image as a function of a line that coincides on the centers of mass for the at least two landmarks.

6. The method of claim 1, wherein calculating displacement includes using a geometric transformation that is applied rigidly between the sensor and synthetic image datasets and a curve fitting model in a region between the at least two landmarks.

7. The method of claim 1, wherein stabilizing images includes minimizing a least mean square error between the centers of mass for corresponding image gradients.

8. The system of claim 1, wherein stabilizing images includes minimizing a least mean square error between corresponding image gradients.

9. A fusion method for combining sensor data with synthetic data to form an integrated image, comprising in combination:

identifying at least two landmarks;

extracting an image gradient from a sensor image dataset for each of the at least two landmarks;

reading a corresponding image synthetic edge from a synthetic image dataset for each of the at least two landmarks;

calculating a center of mass for each of the extracted image gradients and corresponding synthetic edges;

calculating displacement between corresponding image gradients and synthetic edges centered at the calculated centers of mass; and

stabilizing images by minimizing the displacement to form the integrated image.

10. The method of claim 9, wherein calculating displacement includes calculating a displacement vector that represents space shifting and rotation between the extracted image gradient and the corresponding synthetic edge.

11. The method of claim 10, wherein calculating displacement includes estimating the rotation using a rotation matrix.

12. The method of claim 10, wherein calculating displacement includes estimating the rotation as an average of two rotation angles.

13. The method of claim 9, wherein calculating displacement includes estimating an orientation of an image as a function of a line that coincides on the centers of mass for the at least two landmarks.

14. The method of claim 9, wherein calculating displacement includes using a geometric transformation that is applied rigidly between the sensor and synthetic image datasets and a curve fitting model in a region between the at least two landmarks.

15. The method of claim 9, wherein stabilizing images includes minimizing a least mean square error between centers of mass for corresponding image gradients and synthetic edges.

16. The method of claim 9, wherein stabilizing images includes minimizing a least mean square error between corresponding image gradients and synthetic edges.

17. A system for combining sensor data and synthetic data to form an integrated image, comprising in combination:

a synthetic vision system that receives the synthetic data from a database;

a sensor fusion function that receives the sensor data and combines the sensor data if the sensor data originates from more than one sensor;

an image warping function that receives the synthetic data from the synthetic vision system and the sensor data from the sensor fusion function, wherein the image warping function maps the synthetic data into sensor data coordinates;

an image registration function that receives an output from the image warping function and extracts corresponding features from the sensor data and synthetic data, wherein the image registration function matches the extracted features; and

an image fusion function that receives the matched features from the image registration function and generates pixel values based on a weighted average between the two image sources.

18. The system of claim 17, wherein the sensor fusion function performs a selective spatial distribution of millimeter-wave sensor data and a multi-resolution analysis on other received sensor data.

19. The system of claim 17, wherein the image fusion function uses linear superposition to determine a percentage of sensor data and synthetic data to be presented in the integrated image.

20. The system of claim 17, wherein the image fusion function warps pixel values of the synthetic data into a sensor reference using a synthetic domain to represent a global representation of a scene.