System and method for super-resolution imaging from a sequence of color filter array (CFA) low-resolution images

• US 8,666,196 B2
• Filed: 12/20/2010
• Issued: 03/04/2014
• Est. Priority Date: 01/19/2005
• Status: Active Grant

• Current Assignee: The United States Of America As Represented By The Secretary Of The Army
• Original Assignee: The United States Of America As Represented By The Secretary Of The Army
• Inventors: Young, Shiqiong Susan
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US12/973,667
• Publication Number:
  US 20110150331A1
• Time in Patent Office: 1,170 Days
• Field of Search: 382/280, 382/299, 382/321
• US Class Current: 382/299
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: G06T 3/4069 by subpixel displacements

Abstract

A method and system for improving picture quality of color images by combing the content of a plurality of frames of the same subject; comprising: at least one processor; the at least one processor comprising a memory for storing a plurality of frames of a subject; the at least one processor operating to combine the content of plurality of frames of the subject into a combined color image by performing: a process in which at least two multicolored frames are converted to monochromatic predetermined color frames; a gross shift process in which the gross shift translation of one monochromatic predetermined color frame is determined relative to a reference monochromatic predetermined color frame; a subpixel shift process utilizing a correlation method to determine the translational and/or rotational differences of one monochromatic predetermined color frame to the reference monochromatic predetermined color frame to estimate sub-pixel shifts and/or rotations between the frames; and an error reduction process to determine whether the resolution of the resulting combined color image is of sufficient resolution; the error reduction process comprising applying at least one spatial frequency domain constraint and at least one spatial domain constraint to the combined color image to produce at least one high-resolution full color image.

Claims

1. A system for improving picture quality of color images by combining the content of a plurality of frames of the same subject comprising:

at least one processor;

the at least one processor operatively connected to a memory for storing a plurality of frames of a subject;

the at least one processor operating to combine the content of plurality of frames of the subject into a combined color image by performing;

a process in which at least two multicolored frames are converted to monochromatic predetermined color frames;

a gross shift process in which the gross shift translation of one monochromatic predetermined color frame is determined relative to a reference monochromatic predetermined color frame;

a subpixel shift process utilizing a correlation method to determine the translational and/or rotational differences of one monochromatic predetermined color frame to the reference monochromatic predetermined color frame to estimate sub-pixel shifts and/or rotations between the frames; and

an error reduction process to determine whether the resolution of the resulting combined color image is of sufficient resolution;

the error reduction process comprising applying at least one spatial frequency domain constraint and at least one spatial domain constraint to the combined color image to produce at least one high-resolution full color image.

2. The system of claim 1 wherein the process for converting at least two frames to predetermined color frames comprises changing at least two multicolored frames into monochromatic predetermined color frames by substituting pixels of the predetermined color for pixels which are not of the predetermined color to create monochromatic predetermined color frames.

3. The system of claim 2 where the process of substituting pixels of the predetermined color for pixels which are not of the predetermined color comprises keeping the pixels of predetermined color and zeroing the pixels which are not of the predetermined color for each frame, converting the resultant frame into a signal, performing a Fourier transform on the signal, determining the cut-off frequency for the signal, passing the signal through low-pass windowing, and performing an inverse Fourier transform on the signal to obtain the predetermined colored pixel values for every pixel location in the frame and subsequently repeating the process for the plurality of frames to produce a monochromatic color sequence of frames.

4. The system of claim 3 further comprising creating monochromatic color sequence of frames for each of the primary colors in the multicolored frames and assigning the estimated gross shift and estimated subpixel shift obtained using the predetermined color frame sequence to the other color frame sequences to obtain a full color image and wherein the full color image comprises one of (a) green, red, blue or (b) green, red, blue, and white or (c) red, green, blue, and cyan.

5. The system of claim 1 wherein the subpixel shift process comprises estimating the rotation and/or translation of one frame relative to the reference frame of one of a plurality of primary colors.

6. The system of claim 1 wherein the plurality of frames are of a moving subject which is one of a speeding vehicle, an object on a conveyer belt, a face of a person, or nomenclature on clothing.

7. The system of claim 1 wherein the plurality of frames comprise pixels of three primary colors and wherein the error reduction process comprises:

a) estimating translation and/or rotation values for three channels;

b) initializing a processing arrays for each of the three color channels by populating the grid of each processing array using data from a respective one of the three sampled color channels and the translation and rotation estimates for each color channel;

c) applying a 2D Fourier transform to the three processing arrays, respectively, to obtain Fourier transformed processing arrays;

d) determining spatial frequency domain constraints for each of the three processing arrays;

e) applying spatial frequency domain constraints to each of the three the Fourier transformed processing arrays to obtain the constrained spatial frequency domain processing arrays;

f) applying an inverse 2D Fourier transform to the constrained spatial frequency domain processing arrays to obtain inverse processing arrays for each color;

g) applying spatial domain constraints using the estimated shifts and rotations to the inverse processing arrays to obtain the constrained spatial domain processing arrays for each color;

h) checking the error-energy reduction condition;

if the stopping criterion is not satisfied, repeating the steps (c) through (g); and

if the stopping criterion is satisfied then

i) reducing the bandwidth from each of the processing arrays to an output array bandwidth using the Fourier windowing method; and

j) outputting a super-resolved full color image with the desired bandwidth.

8. A system for super-resolving images from color filter array (CFA) low-resolution sequences comprising:

an image processor having an input for inputting multiple CFA images of a scene with sub-pixel translations and rotations;

one of the images of the scene being a reference image; and

at least one memory comprising;

a monochromatic color image constructing algorithm for creating at least one monochromatic color image for each image in the input sequence;

a gross translation estimation algorithm for estimating overall translations of the at least one monochromatic color image with respect to the reference image on the image processor;

a sub-pixel estimation algorithm for estimating the sub-pixel translations and/and or rotations of at least one monochromatic color image with respect to the reference image; and

an error reduction algorithm for applying at least one spatial frequency domain constraint and at least one spatial domain constraint to the CFA low resolution images to produce at least one high-resolution full color image.

9. The system of claim 8 wherein the monochromatic color image constructing algorithm creates a plurality of monochromatic color sequences and wherein the gross translation estimation and subpixel estimation performed on one of monochromatic color sequences are assigned to other monochromatic color sequences.

10. The system of claim 8 wherein the operations performed by the monochromatic color image constructing algorithm comprise obtaining the hexagonal sampled image;

applying 2-D Fourier transform to the hexagonal sampled image to obtain transformed image;

determining cutoff frequency;

applying low-pass windowing generated from the cutoff frequency to the transformed image to create low-pass windowed image; and

applying inverse 2-D Fourier transform to the low-pass windowed image.

11. The system of claim 8 wherein the monochromatic color image constructing algorithm processes at least three color channel sequences and the gross translations and sub-pixel translations and rotations are estimated for at least three color channel sequences.

12. The system of claim 8 wherein the operations performed by the monochromatic color image constructing algorithm comprises obtaining either hexagonal sampled or rectangular sampled images for each monochromatic color, applying 2-D Fourier transform to the sampled images to obtain Fourier transform images;

determining cutoff frequencies;

applying low-pass windowing generated from the cutoff frequencies to the Fourier transformed images to obtain low-pass window images; and

applying inverse 2-D Fourier transform to the low-pass windowed images.

13. The system of claim 12 wherein the monochromatic color image constructing algorithm processes at least three color channel sequences

i. one color is the luminance channel and is obtained using hexagonal sampled images;

ii. two colors are the chrominance colors and are obtained using rectangular sampled images;

and wherein the operations performed by determining cutoff frequencies comprise designing different cutoff frequencies for the luminance channel and chrominance channels.

14. The system of claim 8 wherein the operations performed by the error-reduction algorithm comprise:

a) inputting CFA images and estimated translation and rotation values for three color channels;

b) initializing a processing array for each of the three color channels by populating the grid of each processing array using data from a respective one of the three sampled color channels and the estimated translation and rotation estimates for each color channel;

c) applying a 2D Fourier transform to the three processing arrays, respectively, to obtain Fourier transformed processing arrays;

d) determining spatial frequency domain constraints for each of the three processing arrays;

e) applying spatial frequency domain constraints to each of the three the Fourier transformed processing arrays to obtain the constrained spatial frequency domain processing arrays;

f) applying an inverse 2D Fourier transform to the constrained spatial frequency domain processing arrays to obtain inverse processing arrays for each color;

g) applying spatial domain constraints using the estimated shifts and rotations to the inverse processing arrays to obtain the constrained spatial domain processing arrays for each color;

h) checking the error-energy reduction condition;

if the stopping criterion is not satisfied, repeating the steps (c) through (g); and

if the stopping criterion is satisfied then

i) reducing the bandwidth from each of the processing arrays to an output array bandwidth using the Fourier windowing method; and

j) outputting a super-resolved full color image with the desired bandwidth.

15. The system of claim 8 wherein the at least one spatial frequency domain constraint is applied using the cutoff frequency designed from the hexagonal sampling properties for the green channel;

and using the cutoff frequency designed from the rectangular sampling properties for one of the other color channels.

16. A method of super-resolving CFA images affected by movement including translation and rotation comprising:

inputting frames of low-resolution CFA images of a scene into a processor;

at least one of the frames being affected by movement;

obtaining at least one monochromatic color image for each image in the input frames;

estimating the gross translations and sub-pixel shifts and rotations due to the movement using the monochromatic color images; and

applying an error reduction algorithm utilizing at least one spatial frequency domain constraint and at least one spatial domain constraint to produce a high-resolution full color output image.

17. The method of claim 16 wherein the step of obtaining a monochromatic color image is completed for a plurality of different color monochromatic colors and the estimated gross translations and sub-pixel translations and rotations from the first monochromatic channel are assigned to other monochromatic channels.

18. The method of claim 16 wherein the step of obtaining a monochromatic color image comprises obtaining the hexagonal sampled images;

applying 2-D Fourier transform to each hexagonal sampled portion to obtain a Fourier transform image;

determining cutoff frequency;

applying low-pass windowing generated from the cutoff frequency to each Fourier transformed images to obtain a low pass window image; and

applying inverse 2-D Fourier transform to each low-pass windowed image.

19. The method of claim 16 wherein the steps of applying the error reduction algorithm comprise:

a) inputting CFA images and estimated translation and rotation values for three color channels;

b) initializing a processing arrays for each of the three color channels by populating the grid of each processing array using data from a respective one of the three sampled color channels and the estimated translation and rotation estimates for each color channel;

c) applying a 2D Fourier transform to the three processing arrays, respectively, to obtain Fourier transformed processing arrays;

d) determining spatial frequency domain constraints for each of the three processing arrays;

e) applying spatial frequency domain constraints to each of the three the Fourier transformed processing arrays to obtain the constrained spatial frequency domain processing arrays;

f) applying an inverse 2D Fourier transform to the constrained spatial frequency domain processing arrays to obtain inverse processing arrays for each color;

g) applying spatial domain constraints using the estimated shifts and rotations to the inverse processing arrays to obtain the constrained spatial domain processing arrays for each color;

h) checking the error-energy reduction condition;

if the stopping criterion is not satisfied, repeating the steps (c) through (g); and

if the stopping criterion is satisfied then

i) reducing the bandwidth from each of the processing arrays to an output array bandwidth using the Fourier windowing method; and

j) outputting a super-resolved full color image with the desired bandwidth.

20. The method of claim 19 wherein the predominate color is green and wherein the step of initializing processing arrays comprise:

a) providing the estimated gross translation and sub-pixel shift and rotation of each monochromatic color image with respect to a reference image;

b) calculating the transformed coordinates from the estimated gross translations and sub-pixel shifts and rotation angles for each input monochromatic color image;

c) generating three 2D processing arrays with a sample spacing smaller than the desired high-resolution output image,

d) assigning the known green monochromatic channel values to the closest gross translated and sub-pixel shifted and rotated grid locations of the predetermined processing array in a hexagonal sampling pattern;

e) assigning the known red or blue image values to the closest gross translated and sub-pixel shifted and rotated grid locations of the red or blue processing array in a rectangular sampling pattern; and

f) assigning zeros to other grid locations.

21. The method of claim 16 wherein error reduction algorithm comprises an iterative error energy reduction algorithm using the transformations of each color channel low-resolution image in three upsampled grids to reconstruct the high-resolution three channel full color output, the error-energy reduction algorithm utilizing the low resolution CFA images and their transformations of each color channel on the upsampled grids as the spatial domain constraint and the bandwidth of each color channel as the spatial frequency domain constraint.

22. The method of claim 16 wherein the at least one spatial domain constraint of the error reduction algorithm is the estimated translations and rotations of the low resolution green channel images in a hexagonal sampling pattern and their values;

the estimated translations and rotations of the low-resolution red or blue channel images in a rectangular sampling pattern and their values; and

the spatial frequency domain constraint is the bandwidth determined from cutoff frequencies of hexagonal sampled green channel image and rectangular sampled red or blue channel images.