Dynamic reconstruction of high-resolution video from color-filtered low-resolution video-to-video super-resolution

• US 7,379,612 B2
• Filed: 10/19/2006
• Issued: 05/27/2008
• Est. Priority Date: 12/16/2004
• Status: Active Grant

• Current Assignee: Regents of the University of California (University of California)
• Original Assignee: University of California Santa Cruz (University of California)
• Inventors: Milanfar, Peyman, Farsiu, Sina, Elad, Michael
• Primary Examiner(s): Kassa; Yosef
• Application Number:
  US11/584,400
• Publication Number:
  US 20070071362A1
• Time in Patent Office: 586 Days
• Field of Search: 382/167, 382/169, 382/255, 382/260, 382/284, 382/294, 382/299, 348/241, 348/254, 358/1.2, 358/3.26, 358/3.27
• US Class Current: 382/254
• CPC Class Codes: G06T 3/4076 using the original low-resolution images to iteratively correct the high-resolution images
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60

Abstract

A method is provided of solving the dynamic super-resolution (SR) problem of reconstructing a high-quality set of monochromatic or color superresolved images from low-quality monochromatic, color, or mosaiced frames. The invention includes a joint method for simultaneous SR, deblurring, and demosaicing, this way taking into account practical color measurements encountered in video sequences. For the case of translational motion and common space-invariant blur, the proposed invention is based on a very fast and memory efficient approximation of the Kalman filter (KF). Experimental results on both simulated and real data are supplied, demonstrating the invention algorithms, and their strength.

Claims

1. A method of video-to-video dynamic super-resolution for reducing memory requirements and enhancing processing speed, color and resolution for grayscale and color sequences comprising:

a. using super-resolution, deblurring, and demosaicing simultaneously to incorporate color measurements encountered in video sequences;

b. using a pixel-wise mean-covariance approximate of a Kalman filter for an optimal pilot fused image on a high resolution grid;

c. applying advanced priors to both spatial and color-wise relationships for high quality recovery; and

d. using a non-causal process mode, wherein each said pilot fused image is derived as an optimal estimate incorporating information from all frames in a sequence using smoothed said approximated pixelwise Kalman filter to obtain an efficient algorithm.

2. The method of claim 1, wherein a diagonal covariant matrix is applied to said approximated Kalman filter for enhanced processing speed.

3. The method of claim 1, wherein said data fusion comprises:

a. causally estimating a state vector at time (t≧

1) from a low-resolution frame at time (t≧

1);

b. choosing a high-resolution frame at time (t=0) and said covariance matrix at time (t=0)=∈

²I, whereby ensuring said covariance matrix at time (t) is strictly positive definite at all times;

c. obtaining a low-resolution input image by;

i. setting t=t+1;

ii. applying a motion compensation by computing a forward-shifted matrix at time (t) and a forward shifted covariance matrix at time (t);

iii. updating said covariance matrix at time (t);

iv. updating said state vector at time (t); and

d. repeating step c.

4. The method of claim 1, wherein said image interpolation, deblurring, and demosaicing comprise:

a. using a data fidelity penalty term, wherein said data fidelity penalty term is an L2 norm penalty term to enforce similarities between said pilot fused image and a high-resolution image estimate;

b. using a spatial luminance penalty term, wherein said spatial luminance penalty term is a penalty term to encourage sharp edges in a luminance component to said high-resolution image;

c. using a spatial chrominance penalty term, wherein said spatial chrominance penalty term is a term to encourage smoothness in a chrominance component of said high-resolution image; and

d. using an inter-color dependencies penalty term, wherein said inter-color dependencies penalty term is a term to encourage homogeneity of an edge location and orientation in different color bands.

5. The method according to claim 1, wherein said lower-resolution color images comprise color filtered images, compressed color images, compressed color filtered images, and an image sequence with color artifacts.