Robust reconstruction of high resolution grayscale images from a sequence of low resolution frames

• US 20070217713A1
• Filed: 11/16/2006
• Published: 09/20/2007
• Est. Priority Date: 12/16/2004
• Status: Active Grant

• Current Assignee: Michael Elad, Michael Robinson, Peyman Milanfar, Sina Farsiu
• Original Assignee: Michael Elad, Michael Robinson, Peyman Milanfar, Sina Farsiu
• Inventors: Milanfar, Peyman, Farsiu, Sina, Elad, Michael, Robinson, Michael
• Granted Patent: US 7,477,802 B2
• Time in Patent Office: Days
• US Class Current: 382/299
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution

Abstract

A computer method of creating a super-resolved grayscale image from lower-resolution images using an L₁ norm data fidelity penalty term to enforce similarities between low and a high-resolution image estimates is provided. A spatial penalty term encourages sharp edges in the high-resolution image, the data fidelity penalty term is applied to space invariant point spread function, translational, affine, projective and dense motion models including fusing the lower-resolution images, to estimate a blurred higher-resolution image and then a deblurred image. The data fidelity penalty term uses the L₁ norm in a likelihood fidelity term for motion estimation errors. The spatial penalty term uses bilateral-TV regularization with an image having horizontal and vertical pixel-shift terms, and a scalar weight between 0 and 1. The penalty terms create an overall cost function having steepest descent optimization applied for minimization. Direct image operator effects replace matrices for speed and efficiency.

Claims

1. A method of creating a super-resolved grayscale image from a plurality of lower-resolution images, the method comprising:

a. using a data fidelity penalty term, wherein said data fidelity penalty term is an L₁ norm penalty term to enforce similarities between low-resolution data and a high-resolution image estimate; and

b. using a spatial penalty term, wherein said spatial penalty term is a penalty term to encourage sharp edges in said high-resolution image.

2. The method according to claim 1, wherein said data fidelity penalty term is applied to space invariant point spread function, translational, affine, projective and dense motion models wherein said data fidelity penalty term comprises the steps of:

a. fusing said lower-resolution images to estimate a blurred higher-resolution image; and

b. estimating a deblurred image from said blurred higher-resolution image, wherein said blurred higher-resolution image is a weighted mean of all measurements of a given pixel after zero filling and motion compensation.

3. The method according to claim 1, wherein said data fidelity penalty term uses motion estimation errors comprising said L₁ norm in a likelihood fidelity term.

4. The method according to claim 1, wherein said spatial penalty term uses bilateral-TV regularization.

5. The bilateral-TV regularization of claim 4 comprises an image having a horizontal pixel-shift term, a vertical pixel-shift term, and a scalar weight between 0 and 1.

6. The method according to claim 1, wherein combining said data fidelity penalty term, said spatial penalty term, create an overall cost function.

7. The overall cost function of claim 6, wherein a steepest descent optimization is applied to said overall cost function for minimization.

8. The method according to claim 1, wherein direct image operator effects comprising blur, high-pass filtering, masking, down-sampling, and shift are implemented in place of matrices for process speed and memory efficiency.

9. The method according to claim 1, wherein said lower-resolution images comprise compressed images and uncompressed images.

10. The method according to claim 1, wherein said method is a computer implemented method.