Motion compensated image registration for overlaid/fused video

• US 7,805,020 B2
• Filed: 07/25/2006
• Issued: 09/28/2010
• Est. Priority Date: 07/25/2006
• Status: Active Grant

• Current Assignee: Elbit Systems of America LLC (Elbit Systems Limited)
• Original Assignee: ITT Manufacturing Enterprises, Inc. (ITT, Inc.)
• Inventors: Lynam, Jeff R., Trudeau, Tim K.
• Primary Examiner(s): Bhatnagar; Anand
• Assistant Examiner(s): CHU, RANDOLPH I
• Application Number:
  US11/492,441
• Publication Number:
  US 20080025640A1
• Time in Patent Office: 1,526 Days
• Field of Search: 382/284, 382/294, 382/103, 382/107, 345629-641, 348584-601, 348/154, 348/155, 356/27, 250/214, 250/330
• US Class Current: 382/284
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A system for compensating image misregistration between at least two image sensors includes a first image sensor and a second image sensor which are disposed on a platform and are configured to provide first and second output images, respectively. The system also includes a motion sensor and a processor. The motion sensor senses movement of the platform. The processor calculates a lag time between the first and second image sensors based on first and second processing delay times of the first and second image sensors, respectively. The processor also calculates an image offset based on the lag time and the movement of the platform sensed by the motion sensor and offsets one of the first or second output image with respect to the other one of the first or second output image based on the image offset. A fuser combines the offset image with the non-offset image.

Claims

1. A system for compensating image misregistration between at least two image sensors comprising:

first and second image sensors disposed on a platform and configured to provide first and second output images, respectively;

a motion sensor for sensing movement of the platform; and

a processor configured to calculate an image offset based on the movement of the platform sensed by the motion sensor;

wherein the image offset is calculated by determining (a) horizontal and vertical velocities of the platform, and (b) processing delay time between the first and second output images, and

the processor provides a vector defining a number of pixels in horizontal rows and vertical columns by which the first and second output images are shifted to compensate for the image misregistration between the first and second output images;

wherein the processor is configured to calculate horizontal and vertical image offsets using the following equation;

offset_x,y=υ

_x,yLag_T,

where;

υ

_x,y is a velocity vector of the horizontal and vertical linear velocities,

Lag_T is the lag time, and

offset_x,y is a vector of the calculated horizontal and vertical image offsets.

2. The system according to claim 1 including an image fuser,

wherein the processor is configured to (i) calculate the image offset further based on the lag time and (ii) offset one of the first or second output image with respect to the other one of the first or second output image based on the image offset, and

the fuser is configured to fuse the first and second output images based on the image offset.

3. The system according to claim 1, wherein the processor is configured to calculate horizontal and vertical linear velocities in a plane of the first or second output image based on the sensed movement.

4. The system according to claim 1 including an image fuser configured to fuse the first and second output images, after offsetting the first or second output image.

5. The system according to claim 4, wherein a synchronization signal is provided to the image fuser as a time reference between the first and second output images.

6. The system according to claim 1, wherein the processor is configured to calculate a vertical linear velocity of the first or second output image, using the following equation:

7. A system for compensating image misregistration between at least two image sensors comprising:

first and second image sensors disposed on a platform and configured to provide first and second output images, respectively;

a motion sensor for sensing movement of the platform;

a processor configured to calculate an image offset based on the movement of the platform sensed by the motion sensor;

the motion sensor includes an accelerometer configured to sense horizontal and vertical linear accelerations of the platform; and

an integrator for integrating over time the horizontal and vertical linear accelerations to provide respective horizontal and vertical linear velocities of the platform; and

first and second frame memories configured to store respective first and second previous frames of the first or second output image, and wherein the processor is configured to zero the integrator when the processor determines that the first and second previous frames are spatially correlated to within an error limit.

8. The system according to claim 7 wherein the processor is configured to calculate horizontal and vertical linear velocities of the first or second output image based on the horizontal and vertical linear velocities of the platform.

9. The system according to claim 7, wherein the processor is configured to calculate the horizontal linear velocity of the first or second output image using the following equation:

10. The system according to claim 7, wherein the processor is configured to calculate the vertical linear velocity of the first or second output image using the following equation:

11. The system according to claim 7, wherein the first image sensor includes an optical system configured to provide the first output image, and

the second image sensor includes an electronic image sensor configured to provide the second output image.

12. The system according to claim 11 including an image fuser comprising a digital display and a combining prism, wherein:

the optical system is configured to project onto the combining prism the first output image,

the digital display is configured to project onto the combining prism a light image based upon the second output image, the image offset, and the lag time, and

the combining prism is configured to fuse the first image with the projected light image.

13. A system for compensating image misregistration between at least two image sensors comprising:

first and second image sensors disposed on a platform and configured to provide first and second output images, respectively;

a motion sensor for sensing movement of the platform; and

a processor configured to calculate an image offset based on the movement of the platform sensed by the motion sensor;

wherein the motion sensor includes a rate sensor configured to sense angular velocities of the platform, and

the processor is configured to calculate a horizontal linear velocity of the first or second output image, using the following equation;

14. The system according to claim 13, wherein the first and second image sensors include, respectively, first and second processing delay times, and

the processor is configured to calculate a lag time between the first and second image sensors based on the first and second processing delay times.

15. The system according to claim 14 including an image fuser,

wherein the processor is configured to (i) calculate the image offset further based on the lag time and (ii) offset one of the first or second output image with respect to the other one of the first or second output image based on the image offset, and

the fuser is configured to fuse the first and second output images based on the image offset.

16. The system according to claim 14, wherein the processor is configured to calculate horizontal and vertical linear velocities in a plane of the first or second output image based on the sensed movement.

17. The system according to claim 16, wherein the processor is configured to offset the first or second output image based on the calculated horizontal and vertical linear velocities.

18. The system according to claim 17 including an image fuser configured to fuse the first and second output images, after offsetting the first or second output image.

19. The system according to claim 18, wherein a synchronization signal is provided to the image fuser as a time reference between the first and second output images.

20. The system according to claim 13, wherein the processor is configured to calculate a vertical linear velocity of the first or second output image, using the following equation:

21. The system according to claim 13, wherein the processor is configured to calculate horizontal and vertical image offsets using the following equation:

offset_x,y=υ

_x,yLag_T,

where;

υ

_x,y is a velocity vector of the horizontal and vertical linear velocities,

Lag_T is the lag time, and

offset_x,y is a vector of the calculated horizontal and vertical image offsets.