System and method of super-resolution imaging from a sequence of translated and rotated low-resolution images

• US 7,856,154 B2
• Filed: 10/08/2009
• Issued: 12/21/2010
• Est. Priority Date: 01/19/2005
• Status: Expired due to Fees

• Current Assignee: United States Department of the Army (Government of the United States of America)
• Original Assignee: The United States Of America As Represented By The Secretary Of The Army
• Inventors: Young, Shiqiong Susan
• Primary Examiner(s): Carter; Aaron W
• Application Number:
  US12/576,132
• Publication Number:
  US 20100067822A1
• Time in Patent Office: 439 Days
• Field of Search: 382/280, 382/299, 382/321
• US Class Current: 382/299
• CPC Class Codes: G06T 3/4069 by subpixel displacements

Abstract

A method and system for super-resolving images from low-resolution sequences comprising an image processor having an input for inputting multiple images of a scene with sub-pixel translations and rotations; one of the images of the scene being a reference image; and at least one memory comprising a gross translation estimation algorithm for estimating overall translations of at least one image with respect to the reference image on the image processor and aligning the inputted images according to the gross translation estimates on the image processor; a sub-pixel estimation algorithm for obtaining the sub-pixel shift and rotation of each image with respect to the reference image; and an error reduction algorithm for applying at least one spatial frequency domain constraint and at least one spatial domain constraint to the images to produce a high-resolution image.

Claims

1. A system for super-resolving images from low-resolution sequences comprising:

an image processor having an input for inputting multiple images of a scene with sub-pixel translations and rotations;

one of the images of the scene being a reference image; and

at least one memory comprising;

a gross translation estimation algorithm for estimating overall translations of at least one image with respect to the reference image on the image processor and aligning the inputted images according to the gross translation estimates on the image processor;

a sub-pixel estimation algorithm for obtaining the sub-pixel shift and rotation of each image with respect to the reference image; and

an error reduction algorithm for applying at least one spatial frequency domain constraint and at least one spatial domain constraint to the images to produce at least one high-resolution image.

2. The system of claim 1 wherein the at least one spatial frequency domain constraint comprises the sensor imposed bandwidth.

3. The system of claim 1 wherein the operations performed by the sub-pixel estimation algorithm comprise applying 2-D Fourier transforms to the reference image and a first image;

creating a radially symmetrical window to the Fourier transforms of the first and reference images to obtain windowed images;

constructing the polar transforms of the windowed images to obtain two polar function mappings;

creating the marginal 1-D Fourier transforms with respect to the angle domain for the magnitudes of the two polar function mappings;

forming the cross-correlation function of the two marginal 1-D Fourier transformed images;

estimating the sub-pixel rotation angle θ

between two images from the cross-correlation function;

rotating the second image by the angle −

θ

to form the backward rotated image; and

estimating the sub-pixel shift between the first image and the second backward rotated image.

4. The system of claim 1 wherein the operations performed by the error reduction algorithm comprise:

a) initializing a processing array by populating the grids using the input image values and sub-pixel shift and rotation estimates;

b) applying the 2D Fourier transform to the said processing array;

c) applying spatial frequency domain constraints to the Fourier transformed processing array to obtain the constrained spatial frequency domain processing array;

d) applying the inverse 2D Fourier transform to the constrained spatial frequency domain processing array to obtain the inverse processing array;

e) applying spatial domain constraints using the estimated shifts and rotations to the said inverse processing array to obtain the constrained spatial domain processing array;

f) checking the error-energy reduction condition;

if the stopping criterion is not satisfied, goes back to the step of applying the 2D Fourier transform;

if the stopping criterion is satisfied then

i) reducing the bandwidth from the processing array to the desired output array; and

j) outputting the super-resolved image with the desired bandwidth.

5. The system of claim 4 wherein the step of initializing a processing array comprises:

a) providing the estimated sub-pixel shift and rotation of each image with respect to a reference image;

b) calculating the transformed coordinates from the estimated sub-pixel shifts and rotation angles;

d) generating a 2D processing array with a sample spacing smaller than one of the desired high-resolution output image, that is, a 2D processing array with a larger size than the desired high-resolution output image;

d) assigning the known image values to the closest sub-pixel shifted and rotated grid location of the processing array; and

e) assigning zeros to other grid locations.

6. The system of claim 1 wherein the at least one spatial domain constraint is applied using estimated shifts and rotations by replacing image values at the closest sub-pixel shifted and rotated grid locations of the processing array with the known original low-resolution image values and keeping image values at other grid locations of the processing array.

7. A method of super-resolving images affected by movement including translation and rotation comprising:

inputting frames of low-resolution images of a scene into a processor;

at least one of the frames being affected by movement;

estimating the sub-pixel shifts and rotations among the low resolution image frames due to the movement; and

applying an error reduction algorithm utilizing at least one spatial frequency domain constraint and at least one spatial domain constraint to produce a high-resolution output.

8. The method of claim 7 wherein the high resolution output is a sequence of high-resolution images that are generated from multiple sequences of low-resolution images.

9. The method of claim 7 wherein the step of applying the error reduction algorithm comprise:

a) initializing a processing array by populating the grids using the input image values and sub-pixel shift and rotation estimates;

b) applying the 2D Fourier transform to the said processing array;

c) applying spatial frequency domain constraints to the Fourier transformed processing array to obtain the constrained spatial frequency domain processing array;

d) applying the inverse 2D Fourier transform to the constrained spatial frequency domain processing array to obtain the inverse processing array;

e) applying spatial domain constraints using the estimated shifts and rotations to the said inverse processing array to obtain the constrained spatial domain processing array;

f) checking the error-energy reduction condition;

if the stopping criterion is not satisfied, goes back to the step of applying the 2D Fourier transform;

if the stopping criterion is satisfied then

i) reducing the bandwidth from the processing array to the desired output array; and

j) outputting the high resolution output with the desired bandwidth.

10. The method of claim 9 wherein the step of initializing a processing array comprises:

a) providing the estimated sub-pixel shift and rotation of each image with respect to a reference image;

b) calculating the transformed coordinates from the estimated sub-pixel shifts and rotation angles;

d) generating a 2D processing array with a sample spacing smaller than one of the desired high-resolution output image, that is, a 2D processing array with a larger size than the desired high-resolution output image;

d) assigning the known image values to the closest sub-pixel shifted and rotated grid location of the processing array; and

e) assigning zeros to other grid locations.

11. The method of claim 7 wherein error reduction algorithm comprises an iterative error energy reduction algorithm using the transformations of each low-resolution image in an upsampled grid to reconstruct the high-resolution output, the error-energy reduction algorithm utilizing the low resolution images and their transformations on the upsampled grid as the spatial domain constraint and the bandwidth as the spatial frequency domain constraint.

12. The method of claim 7 wherein the step of estimating the sub-pixel shifts and rotations comprises utilizing a correlation method with a shift and rotation property of a Fourier transform to estimate sub-pixel shifts and rotations among the frames of the image.

13. The method of claim 7 wherein the step of applying an error reduction algorithm comprises treating the spatial samples from low-resolution images as a set of constraints to populate an oversampled processing array which is sampled above the desired output bandwidth, whereby the estimated sub-pixel translations and rotations of the low resolution images and their values constitute a spatial domain constraint and the bandwidth of the high resolution output is the criterion used as a spatial frequency domain constraint.

14. The method of claim 7 wherein the spatial domain constraint of the error reduction algorithm is the estimated sub-pixel translations and rotations of the low resolution images and their values, and the spatial frequency domain constraint is the bandwidth.

15. The method of claim 7 wherein the step of estimating the sub-pixel shifts and rotations comprises utilizing a Fourier-based correlation method to explore the translational and rotational differences of Fourier transforms of at least two images.

16. The method of claim 7 wherein the error reduction algorithm reconstructs a high-resolution output image by applying non-uniform interpolation using the constraints both in the spatial and spatial frequency domains.

17. The method of claim 7 wherein the movement that affects the images is caused by the sensor movement as the images are being captured.

18. The method of claim 17 wherein the sensor is a camera.

19. The method of claim 7 wherein the movement that affects the images is caused by movement of at least one target.