Method of super-resolving images

• US 7,602,997 B2
• Filed: 01/19/2005
• Issued: 10/13/2009
• Est. Priority Date: 01/19/2005
• Status: Active Grant

• Current Assignee: The United States of America As Represented By The Secretary of Agriculture
• Original Assignee: The United States Of America As Represented By The Secretary Of The Army
• Inventors: Young, Shiqiong Susan
• Primary Examiner(s): Carter; Aaron W
• Application Number:
  US11/038,401
• Publication Number:
  US 20060159369A1
• Time in Patent Office: 1,728 Days
• Field of Search: 382/321, 382/299, 382/280
• US Class Current: 382/299
• CPC Class Codes: G06T 3/4069 by subpixel displacements

Abstract

This invention presents a super-resolution image reconstruction from a sequence of aliased imagery. The sub-pixel shifts (displacement) among the images are unknown due to uncontrolled natural jitter of the imager. A correlation method is utilized to estimate sub-pixel shifts between each low resolution aliased image with respect to a reference image. An error-energy reduction algorithm is derived to reconstruct the high-resolution alias-free output image. The main feature of this proposed error-energy reduction algorithm is that we treat the spatial samples from low-resolution images that possess unknown and irregular (uncontrolled) sub-pixel shifts as a set of constraints to populate an over-sampled (sampled above the desired output bandwidth) processing array. The estimated sub-pixel locations of these samples and their values constitute a spatial domain constraint. Furthermore, the bandwidth of the alias-free image (or the sensor imposed bandwidth) is the criterion used as a spatial frequency domain constraint on the over-sampled processing array.

Claims

1. A method of super-resolving images comprising the steps of:

a) inputting multiple undersampled low resolution input images of the same scene with unknown/known and irregular/regular sub-pixel shifts among images into an image processor;

b) applying a gross shift estimation algorithm to the undersampled input low resolution images to obtain the overall shift of each image with respect to a reference image on the image processor;

c) aligning the undersampled input images according to the gross shift estimates on the image processor;

d) applying a sub-pixel shift estimation algorithm on the image processor to the aligned undersampled input images to obtain the sub-pixel shift of each image with respect to a reference image; and

e) applying an error-energy reduction algorithm to the undersampled input low resolution images with the estimated sub-pixel shifts among images to produce a high-resolution (alias-free) output image from the image processor.

20. The method of claim 1 wherein the step of applying a gross shift estimation algorithm to the input low resolution images to obtain the overall shift of each image with respect to a reference image comprises utilizing a Fourier-based correlation method to explore the translational differences of Fourier transforms of at least two images.

2. A method of super-resolving images on an image processor comprising the steps of:

a) providing multiple low resolution input images of substantially the same scene with sub-pixel shifts among images;

b) dividing the input images into sub-patches in the spatial domain, where each sub-patch contains multiple small images;

c) for each sub-patch;

i) applying a gross shift estimation algorithm to obtain an overall gross shift estimate for each small image with respect to a reference small image;

ii) aligning the small images according to the gross shift estimate;

iii) applying a sub-pixel shift estimation algorithm to the aligned small images to obtain the estimated sub-pixel shift of each small image with respect to the reference small image; and

iv) applying an error-energy reduction algorithm to the small images with the estimated sub-pixel shifts to produce a high-resolution (substantially alias-free) sub-patch image; and

d) combining all super-resolved sub-patch images to form a high-resolution output image.

3. The method of claim 2 wherein said step of applying a gross shift estimation algorithm to obtain the overall shift estimate for each small image with respect to a reference small image further includes the steps of:

A) providing two input images, denoting the reference image to be the first input image and the selected image the second input image;

B) applying the Fourier transform to the first input image;

C) applying a low-passing windowing to the first Fourier transformed image to obtain the first low-pass windowed image;

D) applying the Fourier transform to the second input image;

E) applying a low-pass windowing to the second Fourier transformed image to obtain the second low-pass windowed image;

F) obtaining the conjugate of the second low-pass windowed image;

G) correlating the first low-pass windowed image with the second conjugated low-pass windowed image;

H) applying the inverse Fourier transform to the said correlated image;

I) finding the peak of the said inverse Fourier transformed correlated image; and

J) outputting the peak as the spatial overall shift of the selected image with respect to the reference image.

6. The method of claim 2 wherein said step of applying a sub-pixel shift estimation algorithm to obtain the estimated sub-pixel shift of each small image with respect to the reference small image further includes the steps of:

A) providing two input low resolution images, denoting the reference image to be the first input image and the selected image the second input image;

B) applying the Fourier transform to the first image;

C) applying the Fourier transform to the second image;

D) obtaining the conjugate of the second Fourier transformed image;

E) correlating the first Fourier transformed image with the second conjugated Fourier transformed image to obtain the correlated image;

F) applying a upsampling procedure to the said correlated image to obtain the upsampled correlated image;

G) applying a low-pass windowing to the said upsampled correlated image to obtain the low-pass upsampled correlated image;

H) applying the inverse Fourier transform to the said low-pass upsampled correlated image to obtain the inverse upsampled correlated image;

I) finding the peak of the said inverse upsampled correlated image; and

J) outputting the peak as the sub-pixel shift of the selected image with respect to the reference image.

4. A method of super-resolving images on an image processor comprising the steps of:

a) providing multiple low resolution input images of the same scene with unknown/known and irregular/regular sub-pixel shifts among images;

b) applying a gross shift estimation algorithm to the input low resolution images to obtain the overall shift of each image with respect to a reference image comprising the substeps of;

i. providing two input images, denoting the reference image to be the first input image and the selected image the second input image;

ii. applying the Fourier transform to the first input image;

iii. applying the Fourier transform to the second input image;

iv. obtaining the conjugate of the second Fourier transformed image;

v. correlating the first Fourier transformed image with the second conjugated Fourier transformed image;

vi. applying a low-pass windowing to the said correlated image to obtain the low-pass correlated image;

vii. applying the inverse Fourier transform to the said low-pass correlated image to obtain the inverse correlated image;

viii. finding the peak of the said inverse correlated image; and

ix. outputting the peak as the spatial overall shift of the selected image with respect to the reference image

c) aligning the input images according to the gross shift estimates;

d) applying a sub-pixel shift estimation algorithm to the aligned input images to obtain the sub-pixel shift of each image with respect to a reference image; and

e) applying an error-energy reduction algorithm to the input low resolution images with the estimated sub-pixel shifts among images to produce a high-resolution (alias-free) output image.

5. A method of super-resolving images on an image processor comprising the steps of:

a) providing multiple low resolution input images of the same scene with unknown/known and irregular/regular sub-pixel shifts among images;

b) applying a gross shift estimation algorithm to the input low resolution images to obtain the overall shift of each image with respect to a reference image;

c) aligning the input images according to the gross shift estimates;

d) applying a sub-pixel shift estimation algorithm to the aligned input images to obtain the sub-pixel shift of each image with respect to a reference image comprising the substeps of

i) providing two input low resolution images, denoting the reference image to be the first input image and the selected image the second input image;

ii) applying the Fourier transform to the first image;

iii) applying an upsample procedure to the first Fourier transformed image to obtain the first upsampled image;

iv) applying the Fourier transform to the second image;

v) applying an upsampling procedure to the second Fourier transformed image to obtain the second upsampled image;

vi) obtaining the conjugate of the second upsampled image;

vii) correlating the first upsampled image with the second conjugated upsampled image to obtain an upsampled correlated image;

viii) applying a low-pass windowing to the upsampled correlated image to obtain a low-pass upsampled correlated image;

ix) applying the inverse Fourier transform to the low-pass upsampled correlated image to obtain an inverse upsampled correlated image;

x) finding the peak of the inverse upsampled correlated image; and

xi) outputting the peak as the sub-pixel shift of the selected image with respect to the reference image; and

e) applying an error-energy reduction algorithm to the input low resolution images with the estimated sub-pixel shifts among images to produce a high-resolution (alias-free) output image.

7. A method of super-resolving images on an image processor comprising the steps of:

a) providing multiple low resolution input images of the same scene with unknown/known and irregular/regular sub-pixel shifts among images;

b) applying a gross shift estimation algorithm to the input low resolution images to obtain the overall shift of each image with respect to a reference image;

c) aligning the input images according to the gross shift estimates;

d) applying a sub-pixel shift estimation algorithm to the aligned input images to obtain the sub-pixel shift of each image with respect to a reference image; and

e) applying an error-energy reduction algorithm to the input low resolution images with the estimated sub-pixel shifts among images to produce a high-resolution (alias-free) output image, wherein said step of applying an error-energy reduction algorithm further includes the steps of;

i) initializing a processing array by populating the grids using the input image values and sub-pixel shift estimates;

ii) applying the 2D Fourier transform to the said processing array;

iii) applying spatial frequency domain constraints to the Fourier transformed processing array to obtain the constrained spatial frequency domain processing array;

iv) applying the inverse 2D Fourier transform to the constrained spatial frequency domain processing array to obtain the inverse processing array;

v) applying spatial domain constraints to the inverse processing array to obtain the constrained spatial domain processing array;

vi) checking the error-energy reduction condition;

vii) if the stopping criterion is not satisfied, going back to the step of applying the 2D Fourier transform;

viii) if the stopping criterion is satisfied, going to the next step;

ix) reducing the bandwidth from the processing array to the desired output array; and

x) outputting the super-resolved image with the desired bandwidth.

8. The method of claim 7 wherein said step of initializing a processing array further includes the steps of:

A) providing input low resolution images;

B) providing the estimated sub-pixel shift of each image with respect to a reference image;

C) generating a 2D processing array with a sample spacing smaller than one of the desired high-resolution output image, that is, a 2D processing array with a larger size than the desired high-resolution output image;

D) assigning the known image values to each sub-pixel shifted grid location of the processing array; and

E) assigning zeros to other grid locations.

9. The method of claim 7 wherein said step of applying spatial frequency domain constraints further includes the steps of:

A) replacing zeros outside the desired bandwidth; and

B) applying a window function to avoid ripple effect.

10. The method of claim 7 wherein said step of applying spatial domain constraints further includes the steps of:

A) replacing image values at known grid locations of the processing array with the known original low resolution image values; and

B) keeping image values at other grid locations of the processing array.

11. The method of claim 7 wherein said step of checking the error-energy reduction condition further includes the steps of:

A) defining an error-energy using the constrained spatial domain processing array; and

B) checking if the error-energy is less than a threshold.

12. A method of super-resolution reconstruction on an image processor that includes the following steps in the following order:

a) providing multiple low resolution input images with sub-pixel shifts among images;

b) dividing input images into sub-sequences;

c) for each sub-sequence, applying a super-resolution image reconstruction algorithm to generate a high-resolution (alias-free) output image;

d) re-group the output images to form a new sequence;

e) dividing the new sequence into sub-sequences;

f) for each new sub-sequence, applying the super-resolution image reconstruction algorithm to generate a high-resolution (alias-free) output image;

g) checking if the resolution of the desired output image is reached;

h) if the resolution of the desired output image is not reached, going to the step of re-group the said output images to form a new sequence; and

i) if the resolution of the desired output image is reached, outputting the desired high-resolution output image.

13. A method of resolving images taken by a camera subjected to uncontrolled movement, comprising:

inputting frames of low resolution images of a scene into a processor;

at least one of the frames being affected by uncontrolled movement of the camera;

estimating the sub-pixel shift among the image frames due to the movement of the camera using the processor;

applying an error-energy reduction algorithm using the processor to the low-resolution input images utilizing the estimated sub-pixel shift to obtain a high-resolution output image.

14. The method of claim 13 wherein the high resolution output is either a single high resolution image that is generated from a collective sequence of low-resolution images, or a sequence of high-resolution images such as in a video sequence that are generated from multiple sequences of low resolution images.

15. The method of claim 13, wherein the step of estimating the sub-pixel shift utilizes a gross shift estimation algorithm to estimate the overall shift of each frame with respect to a reference frame;

the images of successive frames containing not only sub-pixel shifts but also integer pixel shifts;

the gross shifts among frames being compensated for by realigning the input images; and

estimating the sub-pixel shifts for each frame with respect to a reference frame.

16. The method of claim 13 further comprising the steps of:

utilizing a correlation method to estimate sub-pixel shifts among the frames of the image; and

using the location of each low-resolution image in an upsampled grid in an iterative error-energy reduction algorithm to reconstruct the high-resolution image, the error-energy algorithm utilizing the knowledge of samples of aliased images and their locations on the upsampled array as the spatial domain constraint and the bandwidth as the spatial frequency domain constraint.

17. The method of claim 13 further comprising treating the spatial samples from low-resolution images as a set of constraints to populate an oversampled processing array which is sampled above the desired output bandwidth, whereby the estimated sub-pixel locations of these samples and their values constitute a spatial domain constraint and the bandwidth of the alias free image is the criterion used as a spatial frequency domain constraint on the oversampled processing array.

18. The method of claim 13 wherein the spatial domain constraint of the iterative error-energy reduction algorithm is the estimated sub-pixel locations of these samples and their values and spatial frequency domain constraint of the iterative error-energy reduction algorithm is the bandwidth.

19. The method of claim 13 wherein the step of estimating the sub-pixel shift among the image frames comprises:

i) upsampling the low resolution images by zero padding the Fourier transform of each low resolution image;

the upsampling rate being chosen to form a grid having a larger size than the original image;

ii) correlating two upsampled low resolution images, the location of the peak of the correlation in the spatial domain identifying the relative shift in the upsampled grid from the original image; and

iii) forming a basis of an iterative error-energy reduction algorithm from the estimated sub-pixel locations of each low-resolution image in the upsampled grid to reconstruct the high-resolution (alias-free) image.