Motion compensated image registration for overlaid/fused video

• US 20080025640A1
• Filed: 07/25/2006
• Published: 01/31/2008
• Est. Priority Date: 07/25/2006
• Status: Active Grant

• Current Assignee: Elbit Systems of America LLC (Elbit Systems Limited)
• Original Assignee: ITT Manufacturing Enterprises, Inc. (ITT, Inc.)
• Inventors: Trudeau, Tim K., Lynam, Jeff R.
• Granted Patent: US 7,805,020 B2
• Time in Patent Office: Days
• US Class Current: 382/294
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A system for compensating image misregistration between at least two image sensors includes a first image sensor and a second image sensor which are disposed on a platform and are configured to provide first and second output images, respectively. The system also includes a motion sensor and a processor. The motion sensor senses movement of the platform. The processor calculates a lag time between the first and second image sensors based on first and second processing delay times of the first and second image sensors, respectively. The processor also calculates an image offset based on the lag time and the movement of the platform sensed by the motion sensor and offsets one of the first or second output image with respect to the other one of the first or second output image based on the image offset. A fuser combines the offset image with the non-offset image.

Claims

1. A system for compensating image misregistration between at least two image sensors comprising:

first and second image sensors disposed on a platform and configured to provide first and second output images, respectively;

a motion sensor for sensing movement of the platform; and

a processor configured to calculate an image offset based on the movement of the platform sensed by the motion sensor.

2. The system according to claim 1 wherein the first and second image sensors include, respectively, first and second processing delay times, and

the processor is configured to calculate a lag time between the first and second image sensors based on the first and second processing delay times.

3. The system according to claim 2 including an image fuser,

wherein the processor is configured to (i) calculate the image offset further based on the lag time and (ii) offset one of the first or second output image with respect to the other one of the first or second output image based on the image offset, and

the fuser is configured to fuse the first and second output images based on the image offset.

4. The system according to claim 2, wherein the processor is configured to calculate horizontal and vertical linear velocities in a plane of the first or second output image based on the sensed movement.

5. The system according to claim 4, wherein the processor is configured to offset the first or second output image based on the calculated horizontal and vertical linear velocities.

6. The system according to claim 5 including an image fuser configured to fuse the first and second output images, after offsetting the first or second output image.

7. The system according to claim 6, wherein a synchronization signal is provided to the image fuser as a time reference between the first and second output images.

8. The system according to claim 1, wherein the motion sensor includes a rate sensor configured to sense angular velocities of the platform.

9. The system according to claim 8, wherein the processor is configured to calculate a horizontal linear velocity of the first or second output image, using the following equation:

10. The system according to claim 8, wherein the processor is configured to calculate a vertical linear velocity of the first or second output image, using the following equation:

11. The system according to claim 4, wherein the processor is configured to calculate horizontal and vertical image offsets using the following equation:

offset_x,y=υ

_x,yLag_T,

where;

υ

_x,y is a velocity vector of the horizontal and vertical linear velocities,

Lag_T is the lag time, and

offset_x,y is a vector of the calculated horizontal and vertical image offsets.

12. The system according to claim 1, wherein the motion sensor includes an accelerometer configured to sense horizontal and vertical linear accelerations of the platform.

13. The system according to claim 12 including an integrator for integrating over time the horizontal and vertical linear accelerations to provide respective horizontal and vertical linear velocities of the platform.

14. The system according to claim 13 including first and second frame memories configured to store respective first and second previous frames of the first or second output image, and wherein the processor is configured to zero the integrator when the processor determines that the first and second previous frames are spatially correlated to within an error limit.

15. The system according to claim 14 wherein the processor is configured to calculate horizontal and vertical linear velocities of the first or second output image based on the horizontal and vertical linear velocities of the platform.

16. The system according to claim 13, wherein the processor is configured to calculate the horizontal linear velocity of the first or second output image using the following equation:

17. The system according to claim 13, wherein the processor is configured to calculate the vertical linear velocity of the first or second output image using the following equation:

18. The system according to claim 3, wherein the first image sensor includes an optical system configured to provide the first output image, and

the second image sensor includes an electronic image sensor configured to provide the second output image.

19. The system according to claim 18 including an image fuser comprising a digital display and a combining prism, wherein:

the optical system is configured to project onto the combining prism the first output image,

the digital display is configured to project onto the combining prism a light image based upon the second output image, the image offset, and the lag time, and

the combining prism is configured to fuse the first image with the projected light image.

20. The system according to claim 3, wherein the first and second image sensors provide the first and second output images as electrical signals and the fuser includes fusion electronics configured to fuse the first and second output images, after offsetting the first or second output image.

21. A method for compensating image misregistration between at least two image sensors comprising the steps:

(a) providing first and second image sensors disposed on a platform;

(b) receiving from the first and second image sensors first and second output images, respectively;

(c) sensing movement of the platform; and

(d) offsetting the first or second output image received in step (b) based on the movement sensed in step (c) to provide an image offset.

22. The method according to claim 21 further comprising the steps:

(e) determining first and second processing delay times of the first and second image sensors, respectively; and

(f) calculating a lag time between the first and second processing delay times determined in step (e).

23. The method according to claim 22 further comprising the step:

(g) fusing the first and second output images based on the image offset determined in step (d) and the lag time calculated in step (f).