System and method for robust multi-frame demosaicing and color super resolution

• US 20060279585A1
• Filed: 08/17/2006
• Published: 12/14/2006
• Est. Priority Date: 12/17/2004
• Status: Active Grant

• Current Assignee: University of California Santa Cruz (University of California)
• Original Assignee: University of California Santa Cruz (University of California)
• Inventors: Milanfar, Peyman, Farsiu, Sina, Elad, Michael
• Granted Patent: US 7,940,282 B2
• Time in Patent Office: Days
• US Class Current: 345/694
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening

Abstract

A method of creating a super-resolved color image from multiple lower-resolution color images is provided by combining a data fidelity penalty term, a spatial luminance penalty term, a spatial chrominance penalty term, and an inter-color dependencies penalty term to create an overall cost function. The data fidelity penalty term is an L1 norm penalty term to enforce similarities between raw data and a high-resolution image estimate, the spatial luminance penalty term is to encourage sharp edges in a luminance component to the high-resolution image, the spatial chrominance penalty term is to encourage smoothness in a chrominance component of the high-resolution image, and the inter-color dependencies penalty term is to encourage homogeneity of an edge location and orientation in different color bands. A steepest descent optimization is applied to the overall cost function for minimization by applying a derivative to each color band while the other color bands constant.

Claims

1. A method of creating a super-resolved color image from a plurality of lower-resolution color images, the method comprising:

a. using a data fidelity penalty term, wherein said data fidelity penalty term is an L1 norm penalty term to enforce similarities between raw data and a high-resolution image estimate;

b. using a spatial luminance penalty term, wherein said spatial luminance penalty term is a penalty term to encourage sharp edges in a luminance component to said high-resolution image;

c. using a spatial chrominance penalty term, wherein said spatial chrominance penalty term is a term to encourage smoothness in a chrominance component of said high-resolution image; and

d. using an inter-color dependencies penalty term, wherein said inter-color dependencies penalty term is a term to encourage homogeneity of an edge location and orientation in different color bands.

2. The method according to claim 1, wherein said data fidelity penalty term is applied to space invariant point spread function, translational, affine, projective and dense motion models wherein said data fidelity penalty term comprises the steps of:

a. fusing said lower-resolution images to estimate a blurred higher-resolution image; and

b. estimating a deblurred image from said blurred higher-resolution image, wherein said blurred higher-resolution image is a weighted mean of all measurements of a given pixel after zero filling and motion compensation.

3. The method according to claim 1, wherein said data fidelity penalty term uses motion estimation errors comprising said L1 norm in a likelihood fidelity term.

4. The method according to claim 1, wherein said spatial luminance penalty term uses bilateral-TV regularization.

5. The bilateral-TV regularization of claim 4 comprises a luminance image having a weighted sum of color vectors comprising red vectors, green vectors and blue vectors, a horizontal pixel-shift term, a vertical pixel-shift term, and a scalar weight between 0 and 1.

6. The method according to claim 1, wherein said spatial chrominance penalty term uses regularization based on an L2 norm.

7. The method according to claim 1, wherein said inter-color dependencies penalty term comprises a vector outer product norm of all pairs of neighboring pixels.

8. The method according to claim 1, wherein combining said data fidelity penalty term, said spatial luminance penalty term, said spatial chrominance penalty term, and said inter-color dependencies penalty term creates an overall cost function.

9. The overall cost function of claim 8, wherein a steepest descent optimization is applied to said overall cost function for minimization comprising the steps of:

a. applying a derivative to a first color band while having a second and a third color band held constant;

b. applying a derivative to said second color band while having said first and said third color band held constant; and

c. applying a derivative to said third color band while having said first and said second color band held constant.

10. The method according to claim 1, wherein direct image operator effects comprising blur, high-pass filtering, masking, down-sampling, and shift are implemented in place of matrices for process speed and memory efficiency.

11. The method according to claim 1, wherein said lower-resolution color images comprise color filtered images, compressed color images, compressed color filtered images, and an image sequence with color artifacts.

12. The method according to claim 1, wherein said method is a computer implemented method.