System and method for super-resolution imaging from a sequence of color filter array (CFA) low-resolution images
First Claim
1. A system for improving the quality of color images by combining the content of a plurality of frames of the same subject comprising:
- at least one processor;
the at least one processor operatively connected to a memory for storing a plurality of frames of a subject;
the frames containing imperfect images of the subject due being affected by translation and/or rotational movement;
the at least one processor operating to align and combine the content of plurality of frames of the subject into a combined multicolored image;
the at least one processor operating to convert at least two multicolored frames into monochromatic predetermined color frames;
the at least one processor operating to estimate the optimal orientation by using the at least two monochromatic frames and by estimating optimal alignment of the two monochromatic frames at the frame and subpixel levels;
at the frame level, the at least one processor operating to perform a gross shift process in which the gross shift translation of one monochromatic predetermined color frame is determined relative to another monochromatic predetermined color frame;
at the subpixel level, the at least one processor operating to perform a subpixel shift process utilizing a correlation method to determine the translational and/or rotational differences of one monochromatic predetermined color frame to the other monochromatic predetermined color frame to estimate sub-pixel shifts and/or rotations between the frames;
the at least one processor combining the at least two multicolored frames into a multicolored color image based upon the results of the monochromatic gross shift and subpixel shift processes; and
checking to determine whether the resolution of the resulting combined multicolored image is of sufficient resolution;
the checking process operating to determine the validity of the gross shift and subpixel shift processes to produce at least one high-resolution multicolored image.
1 Assignment
0 Petitions
Accused Products
Abstract
A method and system for improving picture quality of color images by combing the content of a plurality of frames of the same subject; comprising: at least one processor; the at least one processor comprising a memory for storing a plurality of frames of a subject; the at least one processor operating to combine the content of plurality of frames of the subject into a combined color image by performing: a process in which at least two multicolored frames are converted to monochromatic predetermined color frames; a gross shift process in which the gross shift translation of one monochromatic predetermined color frame is determined relative to a reference monochromatic predetermined color frame; a subpixel shift process utilizing a correlation method to determine the translational and/or rotational differences of one monochromatic predetermined color frame to the reference monochromatic predetermined color frame to estimate sub-pixel shifts and/or rotations between the frames; and an error reduction process to determine whether the resolution of the resulting combined color image is of sufficient resolution; the error reduction process comprising applying at least one spatial frequency domain constraint and at least one spatial domain constraint to the combined color image to produce at least one high-resolution full color image.
-
Citations
22 Claims
-
1. A system for improving the quality of color images by combining the content of a plurality of frames of the same subject comprising:
-
at least one processor;
the at least one processor operatively connected to a memory for storing a plurality of frames of a subject;
the frames containing imperfect images of the subject due being affected by translation and/or rotational movement;
the at least one processor operating to align and combine the content of plurality of frames of the subject into a combined multicolored image;
the at least one processor operating to convert at least two multicolored frames into monochromatic predetermined color frames;the at least one processor operating to estimate the optimal orientation by using the at least two monochromatic frames and by estimating optimal alignment of the two monochromatic frames at the frame and subpixel levels; at the frame level, the at least one processor operating to perform a gross shift process in which the gross shift translation of one monochromatic predetermined color frame is determined relative to another monochromatic predetermined color frame; at the subpixel level, the at least one processor operating to perform a subpixel shift process utilizing a correlation method to determine the translational and/or rotational differences of one monochromatic predetermined color frame to the other monochromatic predetermined color frame to estimate sub-pixel shifts and/or rotations between the frames; the at least one processor combining the at least two multicolored frames into a multicolored color image based upon the results of the monochromatic gross shift and subpixel shift processes; and checking to determine whether the resolution of the resulting combined multicolored image is of sufficient resolution; the checking process operating to determine the validity of the gross shift and subpixel shift processes to produce at least one high-resolution multicolored image. - View Dependent Claims (2, 3, 4, 5, 6, 7)
a) estimating translation and/or rotation values for three channels; b) initializing a processing arrays for each of the three color channels by populating the grid of each processing array using data from a respective one of the three sampled color channels and the translation and rotation estimates for each color channel; c) applying a 2D Fourier transform to the three processing arrays, respectively, to obtain Fourier transformed processing arrays; d) determining spatial frequency domain constraints for each of the three processing arrays; e) applying spatial frequency domain constraints to each of the three the Fourier transformed processing arrays to obtain the constrained spatial frequency domain processing arrays; f) applying an inverse 2D Fourier transform to the constrained spatial frequency domain processing arrays to obtain inverse processing arrays for each color; g) applying spatial domain constraints using the estimated shifts and rotations to the said inverse processing arrays to obtain the constrained spatial domain processing arrays for each color; h) checking the condition;
if the stopping criterion is not satisfied, repeating the steps (c) through (g); and
if the stopping criterion is satisfied theni) reducing the bandwidth from each of the processing arrays to an output array bandwidth using the Fourier windowing method; and j) outputting a super-resolved multicolored image with the desired bandwidth.
-
-
8. A system for super-resolving images from color filter array (CFA) low-resolution sequences comprising:
-
at least one processor having an input for inputting multiple CFA images of a scene with sub-pixel translations and rotations;
one of the images of the scene being a reference image; andat least one memory operatively connected to the at least one processor, the at least one processor operating to estimate the optimal orientation by first converting the multicolored frames to monochromatic frames using a monochromatic color image constructing algorithm for creating at least one monochromatic color image for each image in the input sequence; the at least one processor operating to estimate the shift of the inputted CFA images using monochromatic frames and performing estimates of the amount of shift of the image from one image to a reference image at the frame and subpixel levels by using a gross translation estimation algorithm for estimating overall translations of the at least one monochromatic color image with respect to the reference image on the image processor; a sub-pixel estimation algorithm for estimating the sub-pixel translations and/or rotations of at least one monochromatic color image with respect to the reference image; and the at least one processor operating to check and validate the estimates obtained using the gross translation and subpixel processes to produce at least one high-resolution full color image. - View Dependent Claims (9, 10, 11, 12, 13, 14, 15)
i. one color is the luminance channel and is obtained using hexagonal sampled images; ii. two colors are the chrominance colors and are obtained using rectangular sampled images; and wherein the operations performed by determining cutoff frequencies comprise designing different cutoff frequencies for the luminance channel and chrominance channels.
-
-
14. The system of claim 8 wherein the validating and checking operations comprise:
-
a) inputting CFA images and estimated translation and rotation values for three color channels; b) initializing a processing arrays for each of the three color channels by populating the grid of each processing array using data from a respective one of the three sampled color channels and the estimated translation and rotation estimates for each color channel; c) applying a 2D Fourier transform to the three processing arrays, respectively, to obtain Fourier transformed processing arrays; d) determining spatial frequency domain constraints for each of the three processing arrays; e) applying spatial frequency domain constraints to each of the three the Fourier transformed processing arrays to obtain the constrained spatial frequency domain processing arrays; f) applying an inverse 2D Fourier transform to the constrained spatial frequency domain processing arrays to obtain inverse processing arrays for each color; g) applying spatial domain constraints using the estimated shifts and rotations to the said inverse processing arrays to obtain the constrained spatial domain processing arrays for each color; h) checking the condition;
if the stopping criterion is not satisfied, repeating the steps (c) through (g); and
if the stopping criterion is satisfied theni) reducing the bandwidth from each of the processing arrays to an output array bandwidth using the Fourier windowing method; and j) outputting a super-resolved full color image with the desired bandwidth.
-
-
15. The system of claim 8 wherein the validating and checking operations comprise applying at least one spatial frequency domain constraint using the cutoff frequency designed from the hexagonal sampling properties for the green channel;
- and using the cutoff frequency designed from the rectangular sampling properties for one of the other color channels.
-
16. A method of super-resolving CFA images affected by movement including translation and rotation comprising:
-
inputting frames of low-resolution CFA images of a scene into a processor;
at least one of the frames being affected by movement;obtaining at least one monochromatic color image for each image in the input frames; estimating the gross translations and sub-pixel shifts and rotations due to the movement using the monochromatic color images; and checking the validity of the estimated gross translations and subpixel shifts to produce a high-resolution full color output image. - View Dependent Claims (17, 18, 19, 20, 21, 22)
determining cutoff frequency;
applying low-pass windowing generated from the cutoff frequency to each Fourier transformed images to obtain a low pass window image; and
applying inverse 2-D Fourier transform to each low-pass windowed image.
-
-
19. The method of claim 16 wherein the steps of checking and validating operations comprise:
-
a) inputting CFA images and estimated translation and rotation values for three color channels; b) initializing a processing arrays for each of the three color channels by populating the grid of each processing array using data from a respective one of the three sampled color channels and the estimated translation and rotation estimates for each color channel; c) applying a 2D Fourier transform to the three processing arrays, respectively, to obtain Fourier transformed processing arrays; d) determining spatial frequency domain constraints for each of the three processing arrays; e) applying spatial frequency domain constraints to each of the three Fourier transformed processing arrays to obtain the constrained spatial frequency domain processing arrays; f) applying an inverse 2D Fourier transform to the constrained spatial frequency domain processing arrays to obtain inverse processing arrays for each color; g) applying spatial domain constraints using the estimated shifts and rotations to the said inverse processing arrays to obtain the constrained spatial domain processing arrays for each color; h) checking the condition;
if the stopping criterion is not satisfied, repeating the steps (c) through (g); and
if the stopping criterion is satisfied theni) reducing the bandwidth from each of the processing arrays to an output array bandwidth using the Fourier windowing method; and j) outputting a super-resolved full color image with the desired bandwidth.
-
-
20. The method of claim 19 wherein the step of initializing processing arrays comprise:
-
a) providing the estimated gross translation and sub-pixel shift and rotation of each monochromatic color image with respect to a reference image; b) calculating the transformed coordinates from the estimated gross translations and sub-pixel shifts and rotation angles for each input monochromatic color image; c) generating three 2D processing arrays with a sample spacing smaller than the desired high-resolution output image, d) assigning the known green monochromatic channel values to the closest gross translated and sub-pixel shifted and rotated grid locations of the predetermined processing array in a hexagonal sampling pattern; e) assigning the known red or blue image values to the closest gross translated and sub-pixel shifted and rotated grid locations of the red or blue processing array in a rectangular sampling pattern; and f) assigning zeros to other grid locations.
-
-
21. The method of claim 16 wherein the checking operation comprises an iterative error energy reduction algorithm using the transformations of each color channel low-resolution image in three upsampled grids to reconstruct the high resolution three channel full color output, the error-energy reduction algorithm utilizing the low resolution CFA images and their transformations of each color channel on the upsampled grids as the spatial domain constraint and the bandwidth of each color channel as the spatial frequency domain constraint.
-
22. The method of claim 19 wherein spatial domain constraint is the estimated translations and rotations of the low resolution green channel images in a hexagonal sampling pattern and their values;
- the estimated translations and rotations of the low-resolution red or blue channel images in a rectangular sampling pattern and their values; and
the spatial frequency domain constraint is the bandwidth determined from cutoff frequencies of hexagonal sampled green channel image and rectangular sampled red or blue channel images.
- the estimated translations and rotations of the low-resolution red or blue channel images in a rectangular sampling pattern and their values; and
Specification