Methods and systems for suppressing atmospheric turbulence in images

• US 9,811,884 B2
• Filed: 05/22/2015
• Issued: 11/07/2017
• Est. Priority Date: 07/16/2012
• Status: Active Grant

• Current Assignee: Teledyne FLIR LLC (Teledyne Technologies Incorporated)
• Original Assignee: Noiseless Imaging Oy Ltd, FLIR Systems, Inc. (Teledyne Technologies Incorporated)
• Inventors: Foi, Alessandro, Katkovnik, Vladimir, Molchanov, Pavlo, Snchez-Monge, Enrique
• Primary Examiner(s): Le, Peter D
• Application Number:
  US14/720,086
• Publication Number:
  US 20150254813A1
• Time in Patent Office: 900 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

Various techniques are disclosed to suppress distortion in images (e.g., video or still images), such as distortion caused by atmospheric turbulence. For example, similar image blocks from a sequence of images may be identified and tracked along motion trajectories to construct spatiotemporal volumes. The motion trajectories are smoothed to estimate the true positions of the image blocks without random displacements/shifts due to the distortion, and the smoothed trajectories are used to aggregate the image blocks in their new estimated positions to reconstruct the sequence of images with the random displacements/shifts suppressed. Blurring that may remain within each image block of the spatiotemporal volumes may be suppressed by modifying the spatiotemporal volumes in a collaborative fashion. For example, a decorrelating transform may be applied to the spatiotemporal volumes to suppress the blurring in a transform domain, such as by alpha-rooting or other suitable operations on the coefficients of the spectral volumes.

Claims

1. A method comprising:

receiving a plurality of video image frames;

extracting a plurality of image blocks from same or different spatial positions on the different received video image frames along motion trajectories, wherein the motion trajectories includes random displacements due to distortion in the received video image frames;

constructing a plurality of spatiotemporal volumes by grouping the extracted image blocks according to the respective motion trajectories;

smoothing the motion trajectories to suppress the random displacements; and

aggregating the image blocks according to the smoothed trajectories to generate a plurality of processed video image frames, wherein at least some of the distortion is suppressed in the processed video image frames.

2. The method of claim 1, wherein the distortion is due to atmospheric turbulence.

3. The method of claim 1, further comprising:

modifying the spatiotemporal volumes to suppress blurring due to the distortion, wherein aggregating of the image blocks comprises aggregating imaging blocks from the modified spatiotemporal volumes.

4. The method of claim 3, wherein the modifying of the spatiotemporal volumes further comprises:

applying a decorrelating transform to the spatiotemporal volumes to generate corresponding three dimensional (3-D) spectra, wherein each 3-D spectrum comprises a plurality of spectral coefficients for a transform domain representation of a corresponding one of the spatiotemporal volumes;

modifying at least some of the spectral coefficients in each of the 3-D spectra to suppress the blurring due to the distortion; and

applying, to the 3-D spectra, an inverse transform of the decorrelating transform to generate the modified spatiotemporal volumes.

5. The method of claim 4, wherein the modifying of the at least some of the spectral coefficients comprises attenuating temporal-AC coefficients and amplifying temporal-DC coefficients of the 3-D spectra.

6. The method of claim 5, wherein the attenuating of the temporal-AC coefficients and the amplifying of the temporal-DC coefficients are by alpha-rooting the temporal-AC and temporal-DC coefficients.

7. The method of claim 1, wherein the smoothing of the motion trajectories is adaptive to a complexity of the motion trajectories and/or adaptive to a magnitude of the random displacements.

8. The method of claim 1, wherein the smoothing of the motion trajectories comprises determining, by regression, approximate positions of the image blocks without the random displacements.

9. The method of claim 1, wherein the extracting of the plurality of image blocks comprises identifying and tracking similar image blocks from the received video image frames.

10. The method of claim 1, wherein each of the image blocks is a fixed-size patch extracted from a corresponding one of the video image frames.

11. A system comprising:

a video interface configured to receive a plurality of video image frames;

a processor in communication with the video interface and configured to;

extract a plurality of image blocks from same or different spatial positions on the different received video image frames along motion trajectories, wherein the motion trajectories includes random displacements due to distortion in the received video image frames,

construct a plurality of spatiotemporal volumes by grouping the extracted image blocks according to the respective motion trajectories;

smooth the motion trajectories to suppress the random displacement, and

aggregate the image blocks according to the smoothed trajectories to generate a plurality of processed video image frames, wherein at least some of the distortion is suppressed in the processed video image frames; and

a memory in communication with the processor and configured to store the processed video image frames.

12. The system of claim 11, wherein the distortion is due to atmospheric turbulence.

13. The system of claim 11, wherein the processor is further configured to:

modify the spatiotemporal volumes to suppress blurring due to the distortion; and

aggregate the image blocks from the modified spatiotemporal volumes.

14. The system of claim 13, wherein the processor is configured to modify the spatiotemporal volumes by:

applying a decorrelating transform to the spatiotemporal volumes to generate corresponding three dimensional (3-D) spectra, wherein each 3-D spectrum comprises a plurality of spectral coefficients for a transform domain representation of a corresponding one of the spatiotemporal volumes;

modifying at least some of the spectral coefficients in each of the 3-D spectra to suppress the blurring due to the distortion; and

applying, to the 3-D spectra, an inverse transform of the decorrelating transform to generate the modified spatiotemporal volumes.

15. The system of claim 14, wherein the modifying of the at least some of the spectral coefficients comprises attenuating temporal-AC coefficients and amplifying temporal-DC coefficients of the 3-D spectra.

16. The system of claim 15, wherein the attenuating of the temporal-AC coefficients and the amplifying of the temporal-DC coefficients are by alpha-rooting the temporal-AC and temporal-DC coefficients.

17. The system of claim 11, wherein the processor is configured to smooth the motion trajectories by determining, using regression, approximate positions of the image blocks without the random displacements.

18. The system of claim 11, wherein the processor is configured to extract the plurality of image blocks by identifying and tracking similar image blocks from the received video image frames.

19. The system of claim 18, wherein the identifying and tracking of the similar image blocks are based on a multiscale motion estimation.

20. The system of claim 11, further comprising an infrared camera configured to capture thermal images of the scene, wherein the received video image frames comprise the captured thermal images.