Self correcting adaptive low light optical payload

• US 10,341,565 B2
• Filed: 05/10/2016
• Issued: 07/02/2019
• Est. Priority Date: 05/10/2016
• Status: Active Grant

• Current Assignee: Raytheon Company (Rtx Corporation)
• Original Assignee: Raytheon Company (Rtx Corporation)
• Inventors: Robinson, Ian S., Bloomer, John D.
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): Uhl, Lindsay J
• Application Number:
  US15/151,033
• Publication Number:
  US 20170332017A1
• Time in Patent Office: 1,148 Days
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/44504 Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits

Abstract

Methods and systems are disclosed for compensating for image motion induced by a relative motion between an imaging platform and a scene. During an exposure period, frames of the scene may be captured respectively in multiple spectral bands, where one of the spectral bands has a lower light level than the first spectral band, and contemporaneous frames include a nearly identical induced image motion. Image eigenfunctions are utilized to estimate the induced image motion from the higher SNR spectral band, and compensate in each of the multiple bands.

Claims

1. A method for capturing images, the method comprising:

capturing, during an exposure period with at least one sensor on a moving imaging platform, frames of a scene in a first photon-rich spectral band and in a second photon-poor spectral band having a lower light level than the first photon-rich spectral band, the frames captured in the first photon-rich spectral band and the frames captured in the second photon-poor spectral band including identical image motion induced by a relative motion between the imaging platform and the scene, wherein the relative motion is unknown and not calculated;

calculating one or more transformations based on measured changes in inter-frame scenes captured in the first photon-rich spectral band to compensate for the induced image motion, wherein the induced image motion includes at least one of rotation, scale, and anamorphic stretch;

digitally transforming the captured frames of the second photon-poor spectral band with the one or more transformations compensating for the induced image motion in the frames captured in the second photon-poor spectral band to remove effects of the induced image motion; and

summing a plurality of successive compensated frames captured in the second photon-poor spectral band to obtain higher signal to noise ratio (SNR) imagery in the second photon-poor spectral band compared to successive uncompensated frames captured in the second photon-poor spectral band,

wherein capturing frames of the scene comprises interleaving first spectral band exposure times with second spectral band exposure times, the first spectral band exposure times being shorter than the second spectral band exposure times.

2. The method of claim 1, wherein the at least one sensor comprises a multi-band focal plane array (FPA).

3. The method of claim 1, further comprising:

identifying, in the frames captured in at least the first photon-rich spectral band, information representing one or more moving targets;

removing the identified information from the frames captured during the exposure period; and

adding the information to at least one of the digitally transformed captured frames.

4. The method of claim 1, wherein:

capturing frames of the scene further comprises co-registering the frames captured in the first photon-rich spectral band and the second photon-poor spectral band; and

the at least one sensor comprises a plurality of focal plane arrays (FPAs), each FPA in the plurality capturing frames in a distinct spectral band and having a known location relative to each other FPA in the plurality.

5. The method of claim 1, wherein the higher SNR imagery comprises a National Imagery Interpretability Rating Scales nighttime image.

6. The method of claim 1, wherein the first photon-rich spectral band includes an emissive band and the second photon-poor spectral band includes a reflective band.

7. The method of claim 1, wherein:

calculating the one or more transformations further comprises comparing successive frame images captured in the first photon-rich spectral band to compute gradients, and fitting the gradients to one or more image eigenfunctions in order to calculate one or more corresponding eigenfunction coefficients; and

digitally transforming the captured frames comprises applying the one or more image eigenfunctions using the corresponding coefficients to the captured frames, and removing known imaging platform trajectory and sensor pointing motion effects from the captured frames.

8. The method of claim 7, wherein the image eigenfunctions include at least two of linear motion, rotation, scale, anamorphic stretch, skew and jitter.

9. A system configured to capture images, comprising:

a movable imaging platform configured with at least one sensor to capture, during an exposure period, frames of a scene in a first spectral band and in a second spectral band having a lower light level than the first spectral band, the frames captured in the first spectral band and the frames captured in the second spectral band including identical image motion induced by a relative motion between the imaging platform and the scene;

a transformation computation processor configured to calculate one or more eigenfunction coefficients based on a fitting of one or more image eigenfunctions to measured changes in inter-frame scenes captured in the first spectral band to compensate for the induced image motion; and

an image correction processor configured to digitally transform the captured frames of the second spectral band with one or more eigenfunction transformations and the calculated one or more eigenfunction coefficients, in order to compensate for the image distortion resulting from the induced image motion, wherein the image distortion includes at least one of rotation, scale, and anamorphic stretch; and

a fusion processor configured to sum a plurality of successive compensated frames captured in the second spectral band to obtain higher signal to noise ratio (SNR) imagery in the second spectral band compared to successive uncompensated frames captured in the second spectral band,

wherein the at least one sensor captures frames of the scene in first spectral band exposure times interleaved with second spectral band exposure times, the first spectral band exposure times being shorter than the second spectral band exposure times.

10. The system of claim 9, wherein the at least one sensor comprises a multi-band focal plane array (FPA).

11. The system of claim 9, further comprising a mover processor configured to:

identify in the frames captured in the first spectral band information representing one or more moving targets;

remove the identified information from the frames captured during the exposure period; and

optionally add the information to at least one of the digitally transformed captured frames.

12. The system of claim 9, wherein:

the at least one sensor comprises a plurality of FPAs, each FPA in the plurality capturing frames in a distinct spectral band and having a known location relative to each other FPA in the plurality; and

the moving imaging platform includes a registration processor configured to co-register the frames captured in the first spectral band and the second spectral band.

13. The system of claim 9, wherein the higher SNR imagery comprises a NIIRS nighttime image.

14. The system of claim 9, wherein the at least first spectral band includes an emissive band and the at least second spectral band includes a reflective band (SWIR).

15. The system of claim 9, wherein:

the transformation computation processor is further configured to calculate the one or more eigenfunction coefficients by comparing successive frame images captured in the first spectral band to compute gradients, and fitting the gradients to one or more image eigenfunctions in order to estimate the induced motion and pointing error; and

the image correction processor is further configured to apply the one or more image eigenfunctions and eigenfunction coefficients to the captured frames of the second spectral band, and removing known imaging platform trajectory and sensor pointing motion effects from the captured frames of the second spectral band.

16. The system of claim 15, wherein the image eigenfunctions include at least one of linear motion, rotation, scale, anamorphic stretch, skew and jitter.