Method and apparatus for enhancing flash and ambient images

• US 7,443,443 B2
• Filed: 07/28/2005
• Issued: 10/28/2008
• Est. Priority Date: 07/28/2005
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Raskar, Ramesh, Agrawal, Amit
• Primary Examiner(s): Vu; Ngoc-Yen
• Assistant Examiner(s): Quiett; Carramah J
• Application Number:
  US11/191,809
• Publication Number:
  US 20070024742A1
• Time in Patent Office: 1,188 Days
• Field of Search: 348/223.1, 348/224.1, 348/227.1, 348/366, 348/370, 348/371, 362 1- 5, 396/155, 396/164, 396/166, 396/167, 382/199, 382/274, 382/275, 382/278
• US Class Current: 348/370
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors

Abstract

A method and system generate an enhanced output image. A first image is acquired of a scene illuminated by a first illumination condition. A second image is acquired of the scene illuminated by a second illumination condition. First and second gradient images are determined from the first and second images. Orientations of gradients in the first and second gradient images are compared to produce a combined gradient image, and an enhanced output image is constructed from the combined gradient image.

Claims

1. A method for generating an enhanced output image, comprising the steps of:

acquiring an ambient image of a scene illuminated by ambient light;

acquiring a flash image of the scene illuminated with a flash unit;

determining a gradient flash image from the flash image;

determining a gradient ambient image from the ambient image;

combining the gradient flash image and the gradient ambient image to produce a combined gradient image;

constructing an enhanced output image from the combined gradient image;

in which the flash image is due to a flash radiance Φ

scaled by a flash power P, and the ambient image is due to an ambient radiance α

scaled by an exposure time E, and wherein flash gradients in the gradient flash image are ∇

Φ and

ambient gradients in the gradient ambient image are ∇

α

;

and

in which a coherence map for the gradient flash image and the gradient ambient image is

$M=\frac{<br><br>\nabla <br><br>\mathrm{\Phi <br><br>}·<br><br>\nabla <br><br>\mathrm{\alpha <br><br>}<br><br>}{<br><br>\nabla <br><br>\mathrm{\Phi <br><br>}<br><br><br><br>\nabla <br><br>\mathrm{\alpha <br><br>}<br><br>}.$

2. The method of claim 1, in which a gradient coherence model is ∇

Φ

=k∇

α

, where k is a scalar.

3. The method of claim 1, in which a symbol →

denotes projecting, and the combining of the gradient flash image and the gradient ambient image is

∇

{tilde over (Φ

)}=(∇

Φ

′

→

∇

α

)=∇

α

(∇

Φ

′

·

∇

α

)/∥

∇

α

∥

²,

where the vector ∇

{tilde over (Φ

)} is along the vector ∇

Φ and

k is a scalar value.

4. The method of claim 1, in which the combined gradient image is

∇

Φ

*=w_S∇

α

+1(1−

w_S)(M∇

Φ

′

+(1−

M)∇

α

,

wherein w_S are weights.

5. The method of claim 4, in which the weights w_S, are a function of normalized pixel intensities.

6. The method of claim 1, in which a symbol →

denotes projecting, and the combined gradient image is

∇

α

*=w_S∇

α

′

+(1−

w_S)(∇

α

′

→

∇

Φ

),

wherein w_S are weights.

7. The method of claim 1, in which the constructing further comprises:

integrating gradients in the combined gradient image.

8. The method of claim 1, further comprising:

generating a residual image according to ∇

η

_A=(1−

w_S)[∇

α

′

−

(∇

α

′

−

∇

Φ

)],

where w_S are weights, and ∇

η

_A represents noise in the ambient image.

9. The method of claim 1, in which a symbol →

denotes projecting, and further comprising;

adding the gradient flash image and the gradient ambient image to produce an additive gradient image ∇

H, and the combined gradient image is

∇

Φ

*=w_ue∇

H+(1−

w_ue)(∇

H→

∇

α

),

where w_S are weights.

10. The method of claim 1, in which the combined gradient field is

∇

Φ

′

=M ∇

Φ

/β

+(1−

M)∇

α

,

where β

is a linear scaling term.

11. The method of claim 1, in which the flash image is acquired while the scene is also illuminated by the ambient light, and further comprising:

subtracting the ambient image from flash image before determining the gradient flash image.

12. A system for generating an enhanced output image, comprising:

a camera configured to acquire a first image of a scene illuminated by a first illumination condition and a second image of the scene illuminated by a second illumination condition;

means for determining a first gradient image from the first image;

means for determining a second gradient image from the second image;

means for comparing the first gradient image and the second gradient image to produce a combined gradient image;

means for constructing an enhanced output image from the combined gradient image;

wherein the first illumination condition is produced by a flash unit, and the second illumination condition is ambient lighting;

in which the first image is due to a flash radiance Φ

scaled by a flash power P, and the second image is due to an ambient radiance α

scaled by an exposure time E, and wherein flash gradients in the first gradient image are ∇

Φ and

ambient gradients in the second gradient image are ∇

α

;

and

in which a coherence map for the first gradient image and the second gradient image is

$M=\frac{<br><br>\nabla <br><br>\mathrm{\Phi <br><br>}·<br><br>\nabla <br><br>\mathrm{\alpha <br><br>}<br><br>}{<br><br>\nabla <br><br>\mathrm{\Phi <br><br>}<br><br><br><br>\nabla <br><br>\mathrm{\alpha <br><br>}<br><br>}.$

13. A method for generating an enhanced output image, comprising the steps of:

acquiring a first image of a scene illuminated by a first illumination condition;

acquiring a second image of the scene illuminated by a second illumination condition;

determining a first gradient image from the first image;

determining a second gradient image from the second image;

comparing an orientation of gradients in the first gradient image and the second gradient image to produce a combined gradient image;

constructing an enhanced output image from the combined gradient image;

in which the first illumination condition is produced by a flash unit and the second illumination condition is produced by ambient lighting;

in which the first image is due to a flash radiance Φ

scaled by a flash power P, and the second image is due to an ambient radiance α

scaled by an exposure time E, and wherein flash gradients in the first gradient image are ∇

Φ and

ambient gradients in the second gradient image are ∇

α

;

and

wherein a coherence map for the first gradient image and the second gradient image is

$M=\frac{<br><br>\nabla <br><br>\mathrm{\Phi <br><br>}·<br><br>\nabla <br><br>\mathrm{\alpha <br><br>}<br><br>}{<br><br>\nabla <br><br>\mathrm{\Phi <br><br>}<br><br><br><br>\nabla <br><br>\mathrm{\alpha <br><br>}<br><br>}.$

14. The method of claim 13, in which a gradient vector for each pixel in the combined gradient image has a magnitude of a gradient vector of a pixel at a corresponding location in the first gradient image and an orientation of a gradient vector of a pixel at a corresponding location in the second gradient image.

15. The method of claim 13, in which the combining further comprises:

projecting, for each pixel in the first gradient image, a gradient vector onto a gradient vector of a corresponding pixel in the second gradient image to obtain a combined gradient vector for a corresponding pixel in the combined gradient image.