FLIR-to-missile boresight correlation and non-uniformity compensation of the missile seeker

• US 20080112594A1
• Filed: 09/15/2004
• Published: 05/15/2008
• Est. Priority Date: 09/15/2004
• Status: Active Grant

• Current Assignee: Raytheon Company (Rtx Corporation)
• Original Assignee: Raytheon Company (Rtx Corporation)
• Inventors: Williams, Darin S,, Kitchen, Edward N.
• Granted Patent: US 7,463,753 B2
• Time in Patent Office: Days
• US Class Current: 382/107
• CPC Class Codes: F41G 7/007 Preparatory measures taken before the launching of the guided missiles
• CPC Class Codes: F41G 7/2226 comparing the observed data with stored target data, e.g. target configuration data
• CPC Class Codes: F41G 7/2253 Passive homing systems, i.e. comprising a receiver and do not requiring an active illumination of the target
• CPC Class Codes: F41G 7/2293 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 3/7803 Means for monitoring or calibrating
• CPC Class Codes: H04N 25/674 based on the scene itself, e.g. defocusing
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

The present invention provides for simple and streamlined boresight correlation of FLIR-to-missile video. Boresight correlation is performed with un-NUCed missile video, which allows boresight correlation and NUC to be performed simultaneously thereby reducing the time required to acquire a target and fire the missile. The current approach uses the motion of the missile seeker for NUCing to produce spatial gradient filtering in the missile image by differencing images as the seeker moves. This compensates DC non-uniformities in the image. A FLIR image is processed with a matching displace and subtract spatial filter constructed based on the tracked scene motion. The FLIR image is resampled to match the missile image resolution, and the two images are preprocessed and correlated using conventional methods. Improved NUC is provided by cross-referencing multiple measurements of each area of the scene as viewed by different pixels in the imager. This approach is based on the simple yet novel premise that every pixel in the array that looks at the same thing should see the same thing. As a result, the NUC terms adapt to non-uniformities in the imager and not the scene.

Claims

1. A method of providing target handover from a FLIR to a missile tracker, comprising:

receiving missile video from the tracker'"'"'s IR imager;

receiving FLIR video from the FLIR;

performing a boresight correlation of the un-NUCed missile video to the FLIR video to align the IR imager to the FLIR; and

performing non-uniformity compensation (NUC) on the missile video.

2. The method of claim 1, wherein the IR image is mounted on a seeker, boresight correlation being performed by,

using the temporal motion of the seeker over a plurality of frames of missile video to form a first edge map;

measuring scene motion in the missile video;

spatially processing images in the FLIR video in accordance with scene motion to locate at least some of the same edges to form a second edge map; and

correlating the first and second edge maps.

3. The method of claim 2, wherein the first edge map is formed by differencing images in the missile video.

4. The method of claim 3, wherein a first frame of the missile video is subtracted from a second frame to form the first difference image.

5. The method of claim 2, wherein the FLIR video is spatially processed with a displace and subtract filter in accordance with the measured scene motion.

6. The method of claim 1, wherein the non-uniformity compensation is performed by,

registering a temporal sequence of input images from the missile video having different FOVs to create an estimate of an input scene;

backward mapping the estimated input scene to the different FOVs to create an estimate input image for each;

comparing each said estimate input image to the associated input image to estimate pixel-by-pixel response non-uniformity compensation (NUC) terms for the IR imager;

applying the NUC terms to the images to generate compensated images.

7. The method of claim 6, wherein the estimate of the input scene is created by,

adding the registered input images together to create a registered sum image;

accumulating the number of input images that contribute to each pixel in the scene in a registered count image;

dividing the registered sum image by the registered count image to form a registered average image.

9. The method of claim 7, wherein the registered average image is maintained at a higher resolution than the input image.

10. The method of claim 9, wherein a nearest neighbor mapping is used to expand the number of pixels in the FOV of the input image into the registered average image, phase variation in the input images causing aliased frequency components from the mapping to cancel while artifacts that are consistent with beyond-Nyquist frequencies in the input image add constructively and are preserved.

11. The method of claim 6, wherein the estimated input scene is backward mapped to each input image by cutting out a frame in the estimated input scene that corresponds to that input image'"'"'s field of view.

12. The method of claim 11, further comprising keeping track of a variability of the scene pixels in the backward mapping to update the estimated NUC terms, wherein those areas of the estimate of the input scene which are temporally stable and locally flat contribute most heavily to the estimate of the NUC term.

13. The method of claim 6, wherein the NUC terms are estimated by,

differencing respective pairs of input images and estimated input images to form a sequence of unexplained difference images;

performing an Nth order correlation on said unexplained difference images to form NUC error terms; and

combining the NUC error terms with the NUC terms to update the NUC terms.

14. The method of claim 7, wherein said backward mapping comprises convolving the FOV motion trace against the registered average image to form an ideal stacked average that is compared to a stacked average of the input images to form a composite unexplained difference image that is output as the NUC error terms to update the NUC terms, said NUC error terms being correlated again the FOV motion trace and added to the registered sum image to update the registered average image for another iteration on the NUC error terms.

15. The method of claim 6, wherein the steps are repeated on the compensated images to improve the estimate of the NUC terms.

16. The method of claim 6, further comprising:

computing a scene pixel variability map;

for each pixel (i,j) in the scene, determine the number N(i,j) of images to merge based on the variability map;

for each pixel in the scene, selecting the most recent N(i,j) images that contribute to said pixel, and

merging the N(i,j) values from the selected images for each pixel to form a composite image.

17. The method of claim 16, wherein a video sequence of said composite images is displayed.

8. The method of claim 8, further comprising keeping track of a variability of the response of the imager pixels in the forward mapping to update the registered sum image, wherein imager pixels that provide the most consistent response contribute most heavily to the registered sum and registered count images, hence the estimate of the input scene

18. A method of correlating uncompensated IR video to reference video, comprising:

using the temporal motion of the IR imager over a plurality of frames of IR video to form a first edge map;

measuring scene motion in the IR video;

spatially processing images in the reference video in accordance with the scene motion to locate at least some of the same edges to form a second edge map; and

correlating the first and second edge maps.

19. The method of claim 18, wherein the first edge map is formed by differencing images in the missile video.

20. The method of claim 19, wherein a first frame of the IR video is subtracted from a second frame to form the first difference image.

21. The method of claim 18, wherein the reference video is spatially processed with a displace and subtract filter in accordance with the measured scene motion.

22. A method of enhancing video images, comprising:

performing non-uniformity compensation (NUC) on a sequence of input images;

providing a scene pixel stationarity map for each input image from the NUC;

providing a scene motion estimate for each input image from the NUC;

for each compensated image, using the scene pixel stationarity map to select a number N_i of images to merge at each scene pixel;

using the scene motion estimate to select the most recent N(i,j) contributors at each image pixel (i,j); and

creating a composite image by merging the N(i,j) contributors from the compensated images at each image pixel.

23. The method of claim 22, wherein a video sequence of said composite images is displayed.

24. A method of non-uniformity compensation (NUC) for a pixilated IR imager, comprising:

registering a temporal sequence of input images from the IR imager having different FOVs to create an estimate of an input scene;

backward mapping the estimated input scene to the different FOVs to create an estimate input image for each;

comparing each said estimate input image to the associated input image to estimate pixel-by-pixel response non-uniformity compensation (NUC) terms for the IR imager; and

applying the NUC terms to the images to generate compensated images.

25. The method of claim 24, wherein the estimate of the input scene is created by,

adding the registered input images together to create a registered sum image;

accumulating the number of input images that contribute to each pixel in the scene in a registered count image;

dividing the registered sum image by the registered count image to form a registered average image.

26. The method of claim 25, further comprising keeping track of a variability of the response of the imager pixels in the forward mapping to update the registered sum image, wherein imager pixels that provide the most consistent response contribute most heavily to the registered sum and registered count images, hence the estimate of the input scene

27. The method of claim 25, wherein the registered average image is maintained at a higher resolution than the input image.

28. The method of claim 27, wherein a nearest neighbor mapping is used to expand the number of pixels in the FOV of the input image into the registered average image, phase variation in the input images causing aliased frequency components from the mapping to cancel while artifacts that are consistent with beyond-Nyquist frequencies in the input image add constructively and are preserved.

29. The method of claim 24, wherein the estimated input scene is backward mapped to each input image by cutting out a frame in the estimated input scene that corresponds to that input image'"'"'s field of view.

30. The method of claim 29, further comprising keeping track of a variability of the scene pixels in the backward mapping to update the estimated NUC terms, wherein those areas of the estimate of the input scene which are temporally stable and locally flat contribute most heavily to the estimate of the NUC term.

31. The method of claim 24, wherein the NUC terms are estimated by,

differencing respective pairs of input images and estimated input images to form a sequence of unexplained difference images;

performing an Nth order correlation on said unexplained difference images to form NUC error terms; and

combining the NUC error terms with the NUC terms to update the NUC terms.

32. The method of claim 25, wherein said backward mapping comprises convolving a FOV motion trace against the registered average image to form an ideal stacked average that is compared to a stacked average of the input images to form a composite unexplained difference image that is output as the NUC error terms to update the NUC terms, said NUC error terms being correlated again the FOV motion trace and added to the registered sum image to update the registered average image for another iteration on the NUC error terms.

33. The method of claim 24, wherein the steps are repeated on the compensated images to improve the estimate of the NUC terms.

34. A method of non-uniformity compensation (NUC) for a pixilated IR imager, comprising:

capturing a temporal sequence of input images from the IR imager having different FOVs;

compensating each input image with apriori NUC terms if they exist to form compensated images;

tracking a FOV motion across the scene;

registering and adding the input images together to create a registered sum image;

accumulating the number of input images that contribute to each pixel in the scene in a registered count image;

dividing the registered sum image by the registered count image to form a registered average image;

using the FOV motion to backward map the registered average image to the different FOVs to create an estimated input image for each;

differencing respective pairs of compensated images and estimated input images to form a sequence of unexplained difference images;

performing an Nth order correlation on said unexplained difference images to form NUC error terms; and

combining the NUC error terms with the NUC terms from the previous iteration to update the NUC terms.

35. The method of claim 34, further comprising keeping track of a variability of the response of the imager pixels to update the registered sum and registered count images and a variability of scene pixels to weight the Nth order correlation.

36. A method of non-uniformity compensation (NUC) for a pixilated IR imager, comprising:

capturing a temporal sequence of input images from the IR imager having different FOVs;

compensating each input image with apriori NUC terms to form compensated images;

tracking a FOV motion across the scene;

registering and adding the input images together to create a registered sum image;

accumulating the number of input images that contribute to each pixel in the scene in a registered count image;

dividing the registered sum image by the registered count image to form a registered average image;

convolving the FOV motion against the registered average image to form an ideal stacked average;

comparing the ideal stacked average to a stacked average of the compensated images to form a composite unexplained difference image that is output as the NUC error terms to update the NUC terms;

correlating said NUC error terms against the FOV motion trace to form a registered sum error; and

adding the registered sum error to the registered sum image to update the registered average image for another iteration on the NUC error terms.