System and method for robust multi-frame demosaicing and color super resolution

• US 7,940,282 B2
• Filed: 08/17/2006
• Issued: 05/10/2011
• Est. Priority Date: 12/17/2004
• Status: Active Grant

• Current Assignee: University of California Santa Cruz (University of California)
• Original Assignee: University of California Santa Cruz (University of California)
• Inventors: Farsiu, Sina, Elad, Michael, Milanfar, Peyman
• Primary Examiner(s): Sajous; Wesner
• Application Number:
  US11/506,246
• Publication Number:
  US 20060279585A1
• Time in Patent Office: 1,727 Days
• Field of Search: 348/223, 348/225, 348/254, 348/273, 348/526, 348/533, 348/538, 348606-607, 348615-616, 348/620, 348/625, 358/504, 358/512, 358515-518, 358/520, 358/525, 358/461, 358/448, 382162-163, 382166-167, 382254-255, 382263-269, 382/274, 382299-300, 345/581, 345/589, 345/597, 345/600, 345/606, 345/611, 345616-619, 345/639, 345/643
• US Class Current: 345/606
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening

Abstract

A method of creating a super-resolved color image from multiple lower-resolution color images is provided by combining a data fidelity penalty term, a spatial luminance penalty term, a spatial chrominance penalty term, and an inter-color dependencies penalty term to create an overall cost function. The data fidelity penalty term is an L1 norm penalty term to enforce similarities between raw data and a high-resolution image estimate, the spatial luminance penalty term is to encourage sharp edges in a luminance component to the high-resolution image, the spatial chrominance penalty term is to encourage smoothness in a chrominance component of the high-resolution image, and the inter-color dependencies penalty term is to encourage homogeneity of an edge location and orientation in different color bands. A steepest descent optimization is applied to the overall cost function for minimization by applying a derivative to each color band while the other color bands constant.

Claims

1. A method of creating a super-resolved color image from a plurality of lower-resolution color images, the method comprising:

a. enforcing by a computer similarities between raw data and a high-resolution image estimate, wherein said enforcing uses a data fidelity penalty term, wherein said data fidelity penalty term is an L1 norm penalty term;

b. encouraging by a computer sharp edges in a luminance component to said high-resolution image, wherein said encouraging uses a spatial luminance penalty term;

c. encouraging by a computer smoothness in a chrominance component of said high-resolution image, wherein said encouraging uses a spatial chrominance penalty term;

d. encouraging by a computer homogeneity of an edge location and orientation in different color bands, wherein said encouraging uses an inter-color dependencies penalty term; and

e. a super-resolved color image is displayed on an computer monitor.

2. The method according to claim 1, wherein said data fidelity penalty term is applied to space invariant point spread function, translational, affine, projective and dense motion models wherein said data fidelity penalty term comprises the steps of:

a. estimating by a computer a blurred higher-resolution image, wherein said estimating comprises fusing said lower-resolution images; and

b. estimating by a computer a deblurred image from said blurred higher-resolution image, wherein said blurred higher-resolution image is a weighted mean of all measurements of a given pixel after zero filling and motion compensation.

3. The method according to claim 1, wherein said data fidelity penalty term uses motion estimation errors comprising said L1 norm in a likelihood fidelity term.

4. The method according to claim 1, wherein said spatial luminance penalty term uses bilateral-TV regularization.

5. The method according to claim 4, wherein said bilateral TV regularization comprises a luminance image having a weighted sum of color vectors comprising red vectors, green vectors and blue vectors, a horizontal pixel-shift term, a vertical pixel-shift term, and a scalar weight between 0 and 1.

6. The method according to claim 1, wherein said spatial chrominance penalty term uses regularization based on an L2 norm.

7. The method according to claim 1, wherein said inter-color dependencies penalty term comprises a vector outer product norm of all pairs of neighboring pixels.

8. The method according to claim 1, wherein using said computer to combine said data fidelity penalty term, said spatial luminance penalty term, said spatial chrominance penalty term, and said inter-color dependencies penalty term creates an overall cost function.

9. The method according to claim 8, wherein said overall cost function comprises a steepest descent optimization, wherein said steepest optimization is applied for minimization by said computer to said overall cost function comprising the steps of:

a. applying by said computer a derivative to a first color band while having a second and a third color band held constant;

b. applying by said computer a derivative to said second color band while having said first and said third color band held constant; and

c. applying by said computer a derivative to said third color band while having said first and said second color band held constant.

10. The method according to claim 1, wherein direct image operator effects comprising blur, high-pass filtering, masking, down-sampling, and shift are implemented by said computer in place of matrices for process speed and memory efficiency.

11. The method according to claim 1, wherein said lower-resolution color images comprise color filtered images, compressed color images, compressed color filtered images, and an image sequence with color artifacts.

12. The method according to claim 1, wherein said method is a computer implemented method.