Method and apparatus for enhancing flash and ambient images

• US 20070024742A1
• Filed: 07/28/2005
• Published: 02/01/2007
• Est. Priority Date: 07/28/2005
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Raskar, Ramesh, Agrawal, Amit
• Granted Patent: US 7,443,443 B2
• Time in Patent Office: Days
• US Class Current: 348/370
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors

Abstract

A method and system generate an enhanced output image. A first image is acquired of a scene illuminated by a first illumination condition. A second image is acquired of the scene illuminated by a second illumination condition. First and second gradient images are determined from the first and second images. Orientations of gradients in the first and second gradient images are compared to produce a combined gradient image, and an enhanced output image is constructed from the combined gradient image.

Claims

1. A method for generating an enhanced output image, comprising the steps of:

acquiring an ambient image of a scene illuminated by ambient light;

acquiring a flash image of the scene illuminated with a flash unit;

determining a gradient flash image from the flash image;

determining a gradient ambient image from the ambient image;

combining the gradient flash image and the gradient ambient image to produce a combined gradient image; and

constructing an enhanced output image from the combined gradient image.

2. The method of claim 1, in which the flash image is due to a flash radiance Φ

scaled by a flash power P, and the ambient image is due to an ambient radiance α

scaled by an exposure time E, and in which flash gradients in the gradient flash image are ∇

Φ and

ambient gradients in the gradient ambient image are Φ

α

.

3. The method of claim 2, in which a coherence map for the gradient flash image and the gradient ambient image is

$M=\frac{}{}$

∇

Φ

·

∇

α





∇

Φ

⁢



∇

α



.

4. The method of claim 2, in which a gradient coherence model is ∇

Φ

=k∇

α

, where k is a scalar.

5. The method of claim 2, in which a symbol →

denotes projecting, and the combining of the gradient flash image and the gradient ambient image is

∇

{tilde over (Φ

)}=(∇

Φ

′

→

∇

α

)=∇

α

(∇

Φ

′

·

∇

α

)/∥

∇

α

∥

²,

where the vector ∇

{tilde over (Φ

)} is along the vector ∇

Φ and

k is a scalar value.

6. The method of claim 2, in which the combined gradient image is

∇

Φ

*=w_S∇

α

+1(1−

w_S)(M∇

Φ

′

+(1−

M)∇

α

,

wherein w_S are weights.

7. The method of claim 6, in which the weights w_S, are a function of normalized pixel intensities.

8. The method of claim 2, in which a symbol →

denotes projecting, and the combined gradient image is

∇

α

*=w_S∇

α

′

+(1−

w_S)(∇

α

′

→

∇

Φ

),

wherein w_S are weights.

9. The method of claim 1, in which the constructing further comprises:

integrating gradients in the combined gradient image.

10. The method of claim 2, further comprising:

generating a residual image according to ∇

η

_A=(1−

w_S)[∇

α

′

−

(∇

α

′

−

∇

Φ

)],

where w_S are weights, and ∇

η

_A represents noise in the ambient image.

11. The method of claim 2, in which a symbol denotes projecting, and further comprising:

adding the gradient flash image and the gradient ambient image to produce an additive gradient image ∇

H, and the combined gradient image is

∇

Φ

*=w_ue∇

H30 (1−

w_ue)(∇

H→

∇

α

),

where w_S are weights.

12. The method of claim 2, in which the combined gradient field is

∇

Φ

′

=M ∇

Φ

/β

+(1−

M)∇

α

,

where β

is a linear scaling term.

13. The method of claim 1, in which the flash image is acquired while the scene is also illuminated by the ambient light, and further comprising:

subtracting the ambient image from flash image before determining the gradient flash image.

14. A system for generating an enhanced output image, comprising:

a camera configured to acquire a first image of a scene illuminated by a first illumination condition and a second image of the scene illuminated by a second illumination condition;

means for determining a first gradient image from the first image;

means for determining a second gradient image from the second image;

means for comparing the first gradient image and the second gradient image to produce a combined gradient image; and

means for constructing an enhanced output image from the combined gradient image.

15. The system of claim 14, in which the first illumination condition is produced by a flash unit, and the second illumination condition is ambient lighting.

16. A system for generating an enhanced output image, comprising:

a camera configured to acquire an ambient image of a scene illuminated by ambient light and a flash image of the scene illuminated with a flash unit;

means for determining a gradient flash image from the flash image;

means for determining a gradient ambient image from the ambient image;

means for combining the gradient flash image and the gradient ambient image to produce a combined gradient image; and

means for constructing an enhanced output image from the combined gradient image.

17. A method for generating an enhanced output image, comprising the steps of:

acquiring a first image of a scene illuminated by a first illumination condition;

acquiring a second image of the scene illuminated by a second illumination condition;

determining a first gradient image from the first image;

determining a second gradient image from the second image;

comparing an orientation of gradients in the first gradient image and the second gradient image to produce a combined gradient image; and

constructing an enhanced output image from the combined gradient image.

18. The method of claim 17, in which a gradient vector for each pixel in the combined gradient image has a magnitude of a gradient vector of a pixel at a corresponding location in the first gradient image and an orientation of a gradient vector of a pixel at a corresponding location in the second gradient image.

19. The method of claim 17, in which the combining further comprises:

projecting, for each pixel in the first gradient image, a gradient vector onto a gradient vector of a corresponding pixel in the second gradient image to obtain a combined gradient vector for a corresponding pixel in the combined gradient image.

20. The method of claim 17, in which the first illumination condition is produced by a flash unit and the second illumination condition is produced by ambient lighting.