Apparatus and method for generating a fully focused image by using a camera equipped with a multi-color filter aperture

• US 8,836,765 B2
• Filed: 11/07/2011
• Issued: 09/16/2014
• Est. Priority Date: 11/05/2010
• Status: Expired due to Fees

• Current Assignee: Industry-University Cooperation Foundation Hanyang University
• Original Assignee: Industry-University Cooperation Foundation Hanyang University
• Inventors: Paik, Joon-Ki, Kim, Sang-Jin, Lee, Eun-Sung
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): WILLIAMS, JEFFERY A
• Application Number:
  US13/290,930
• Publication Number:
  US 20120113227A1
• Time in Patent Office: 1,044 Days
• Field of Search: 348/46, 348/135
• US Class Current: 348/46
• CPC Class Codes: H04N 23/67 Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 23/84 for processing colour signals

Abstract

Provided are an apparatus and method for generating a fully focused image. A depth map generation unit generates a depth map of an input image obtained by a multiple color filter aperture (MCA) camera. A channel shifting & alignment unit extractes subimages which include objects with same focal distance based on the depth map, and performing color channel alignment and removing out-of-focus blurs for each subimages obtained from the depth map. An image fusing unit fuses the subimages to generate a fully focused image.

Claims

1. An apparatus for generating a fully focused image, the apparatus comprising:

a depth map generation unit generating a depth map of an input image obtained by a multiple color filter aperture (MCA) camera;

a channel shifting &

alignment unit extracting subimages which include objects with same focal distance based on the depth map, and performing color channel alignment and removing out-of-focus blurs for each subimages obtained from the depth map; and

an image fusing unit fusing the subimages to generate a fully focused image.

2. The apparatus of claim 1, wherein the depth map generation unit generates the depth map by establishing a set of pixel colors with hypothesized disparity d as Equation 1 shown below, and searches for d that minimizes an abstract disparity map represented by following Equation 2:

P₁(x,y;

d)={(I_(R)(s+d,t),I_(G)(s,t−

d),I_(B)(s−

d,t))|(s,t)ε

w(x,y)} 

 

Equation 1

L(x,y;

d)=λ

₀λ

₁λ

₂/σ

_r²σ

_g²σ

_b² 

 

Equation 2

here, I_(R), I_(G), and I_(B) represent red, green, and blue color channel images, respectively, (s+d,t), (s,t−

d), and (s−

d,t) are shift vectors for each color channel images, λ

₀, λ

₁, and λ

₂ represent the eigenvalues of the covariance matrix Σ

of the color distribution P₁(x,y;

d), respectively, w(x,y) is a local window, and σ

_r², σ

_g², and σ

_b² are the diagonal elements of the covariance matrix Σ

.

3. The apparatus of claim 2, wherein the depth map generation unit generates the depth map by combining the abstract disparity map and a mean-shift segmentation result.

4. The apparatus of claim 1, further comprising:

a smoothing unit smoothing the fully focused image to enhance image quality.

5. A method for generating a fully focused image, the method comprising:

(a) generating a depth map of an input image obtained by a multiple color filter aperture (MCA) camera;

(b) extracting subimages which include objects with same focal distance based on the depth map;

(c) performing color channel alignment and removing out-of-focus blurs for each subimages obtained from the depth map; and

(d) fusing the subimages to generate a fully focused image.

6. The method of claim 5, wherein step (a) comprises:

(a1) establishing a set of pixel colors with hypothesized disparity d as Equation 1 shown below; and

(a2) searching for d that minimizes an abstract disparity map represented by following Equation 2;

P₁(x,y;

d)={(I_(R)(s+d,t),I_(G)(s,t−

d),I_(B)(s−

d,t))|(s,t)ε

w(x,y)} 

 

Equation 1

L(x,y;

d)=λ

₀λ

₁λ

₂/σ

_r²σ

_g²σ

_b² 

 

Equation 2

here, I_(R), I_(G), and I_(B) represent red, green, and blue color channel images, respectively, (s+d,t), (s,t−

d), and (s−

d,t) are shift vectors for each color channel images, λ

₀, λ

₁, and λ

₂ represent the eigenvalues of the covariance matrix Σ

of the color distribution P₁(x,y;

d), respectively, w(x,y) is a local window, and σ

_r², σ

_g², and σ

_b² are the diagonal elements of the covariance matrix Σ

.

7. The method of claim 6, wherein step (a) further comprises (a3) generating the depth map by combining the abstract disparity map and a mean-shift segmentation result.

8. The method of claim 5, further comprising:

(e) smoothing the fully focused image to enhance image quality.

9. A non-transitory computer readable medium storing a program for executing the method for generating a fully focused image according to claim 5 in a computer.