Method of restoring and reconstructing super-resolution image from low-resolution compressed image

• US 20050019000A1
• Filed: 06/25/2004
• Published: 01/27/2005
• Est. Priority Date: 06/27/2003
• Status: Abandoned Application

• Current Assignee: Sungjin C & C Limited, Yonsei University
• Original Assignee: Sungjin C & C Limited, Yonsei University
• Inventors: Kang, Moon Gi, Lim, In-Keon, Park, Sung Cheol
• Application Number:
  US10/875,218
• Publication Number:
  US 20050019000A1
• Time in Patent Office: Days
• US Class Current: 386/209
• CPC Class Codes: G06T 3/4069 by subpixel displacements
• CPC Class Codes: G06T 3/4084 in the transform domain, e.g. fast Fourier transform [FFT] domain scaling

Abstract

Provided is a method of restoring and/or reconstructing a super-resolution image from low-resolution images compressed in a digital video recorder (DVR) environment. The present invention can remove a blur of a video sequence, caused by optical limitations due to a miniaturized camera of a digital video recorder monitoring system, a limitation of spatial resolution due to an insufficient number of pixels of a CCD/CMOS image sensor, and noises generated during image compression, transmission and storing processes, to restore high-frequency components of low-resolution images (for example, the face and appearance of a suspect or numbers of a number plate) to reconstruct a super-resolution image. Consequently, an interest part of a low-resolution image stored in the digital video recorder can be magnified to a high-resolution image later, and the effect of an expensive high-performance camera can be obtained from an inexpensive low-performance camera.

Claims

1. A method of restoring super-resolution (SR) image having a size of L₁N₁×

L₂N₂ from P low-resolution (LR) images, each of which has a size of N₁×

N₂, comprising steps of;

modeling the quantization noise of DCT coefficients for each LR image (which is divided into a plurality of independent blocks, discrete-cosine-transformed and quantized) as a random variable having a Gaussian distribution; and

estimating sub-pixel shifts between the P LR images and a reference image, which is chosen among the P low-resolution images, by obtaining a least mean square of a motion parameter between the reference image and the other images through Taylor'"'"'s series expansion

wherein a smoothing constraint representing prior information about the SR image is modeled as a non-stationary Gaussian distribution to apply an adaptive smoothing constraint, which makes the mean of noises zero, and thereby a compression noise is removed while the contour of the image is preserved.

2. The method as set forth in claim 1, wherein the k-th LR image y_k among the P low-resolution images is modeled by the following equation.

y_k=DB_kM_kx+n_k, k=1,2, . . . ,p

(Here, M_k is a geometrical warping matrix representing a relative shift, B_k is a matrix representing a blur, D is a matrix representing undersampling from SR image to LR image, n_k represents noise including compression noise, and x represents the SR image.)

3. The method as set forth in claim 1, comprising steps of:

(a) magnifying one of the P LR images by interpolation, followed by setting the magnified one as an initial SR image xⁿ;

(b) blurring and down-sampling an image which is obtained by performing the registration on the SR image xⁿ by an estimated motion parameter value of the k-th LR image y_k, followed by calculating a image difference between the blurred/down-sampled image and the k-th LR image y_k;

(c) estimating a one-step correlation parameter in the first-order Markov process for each block of the image difference, followed by multiplying the one-step correlation parameter by a covariance matrix, and by up-sampling and re-blurring the resultant image;

(d) performing the inverse-registration on the resultant image of the step (c) by an amount of the estimated motion parameter value of y_k;

(e) calculating a normalization function. α

_k(x);

(f) calculating a difference between the SR image xⁿ and a nonstationary mean of the SR image, {overscore (x)}, followed by multiplying the resultant image difference by α

_k(x);

(g) obtaining a difference image between the image obtained in the step (d) and the image obtained in the step (f);

(h) executing the steps (a) through (g) for each of the LR images (k=1, . . . , p), followed by summing up the resultant image differences;

(i) multiplying the resultant image of the step (h) by a convergence rate control parameter, followed by adding the high-resolution image xⁿ to the multiplied result to obtain a new image xⁿ⁺¹; and

(j) repeating the steps (a) through (i) until xⁿ⁺¹ converges to xⁿ to obtain the SR image

4. The method as set forth in claim 3, wherein the compression noise is represented by a vector n, which is lexicographically arranged in an arbitrary block of an image, to model a probability density function of a quantization noise in a DCT domain as

5. The method as set forth in claim 4, wherein the inverse matrix R_n⁻

1 of the covariance matrix is modeled as a matrix having a DCT basis function as an eigenvector.

6. The method as set forth in claim 3, wherein the one-step correlation parameter is estimated in each DCT block using a biased sample operator.

7. The method as set forth in claim 1, wherein the motion estimation parameter R_k is represented by R_k=M⁻

1V_k.

8. The method as set forth in claim 1, wherein, when the compression noise is represented by the lexicographically arranged vector n, the inverse matrix of the covariance matrix representing correlation of n is modeled as a kronecker product of tridiagonal Jacobi matrix having the DCT basis function as an eigenvector.

9. The method as set forth in claim 1, wherein the smoothing constraint is

10. The method as set forth in claim 1, wherein the method further comprises a step of controlling balance between image fidelity and the smoothing constraint using the following equation.