Method of capturing high dynamic range images with objects in the scene

• US 8,724,921 B2
• Filed: 05/05/2008
• Issued: 05/13/2014
• Est. Priority Date: 05/05/2008
• Status: Active Grant

• Current Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Original Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Inventors: Jin, Elaine W.
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): TORRES, JOSE
• Application Number:
  US12/115,022
• Publication Number:
  US 20090274387A1
• Time in Patent Office: 2,199 Days
• Field of Search: 382/254, 382/274, 382/284, 348/362
• US Class Current: 382/274
• CPC Class Codes: G03B 7/00 Control of exposure by setting shutters, diaphragms or filters, separately or conjointly control of exposure in television cameras by means of circuitry influencing the optical part of the camera H04N23/75
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/75 by influencing optical camera components

Abstract

Methods, image processors and imaging devices for capturing a high dynamic range (HDR) image. Multiple images of a scene are captured at respectively different exposure settings. A further image of an object placed in the scene is captured at one exposure setting. A first radiance image is formed from the multiple images. A second radiance image is formed from the further image. The first radiance image and the second radiance image are merged to form the HDR image.

Claims

1. A method of capturing a high dynamic range (HDR) image, comprising:

capturing multiple images of a scene at respectively different exposure settings;

capturing a further image of an object placed in the scene at one exposure setting;

forming a first radiance image from the multiple images;

forming a second radiance image from the further image; and

merging the first radiance image and the second radiance image to form the HDR image, wherein the HDR image has a first region that includes the object and is formed from the second radiance image, wherein the HDR image has a second region that excludes the object and is formed from the first radiance image, and wherein the first region of the HDR image has a lower dynamic range than the second region of the HDR image.

2. The method according to claim 1, the step of capturing the multiple images including using a predetermined range of exposure settings.

3. The method according to claim 1, the step of capturing the further image including:

capturing a test image of the object in the scene; and

selecting the one exposure setting based on the captured test image.

4. The method according to claim 3, wherein the one exposure setting is selected to capture a predetermined dynamic range for the object.

5. The method according to claim 1, wherein the first radiance image includes a high dynamic range.

6. The method according to claim 5, wherein an illumination range associated with the high dynamic range is greater than about 90 dB.

7. The method according to claim 1, wherein the second radiance image includes a low dynamic range.

8. The method according to claim 7, wherein an illumination range associated with the low dynamic range is less than about 50 dB.

9. The method according to claim 1, the step of forming the first radiance image including:

generating a respective plurality of radiance images for each of the multiple images based on the corresponding exposure setting; and

combining the plurality of radiance images to form the first radiance image.

10. The method according to claim 1, wherein the scene is slowly varying with time and the object varies faster than the scene.

11. The method according to claim 1, wherein the multiple exposure times are adjusted based on one or more of an exposure time, an F-number or a shutter speed.

12. The method according to claim 1, the step of merging the first and second radiance images comprising:

forming a masked image by subtracting one of the multiple images from the further image;

forming an inverse masked image from an inverse of the masked image;

applying the inverse masked image to the first radiance image;

applying the masked image to the second radiance image; and

merging the masked first and second radiance images to form the HDR image.

13. The method according to claim 12, the step of forming the masked image including selecting one of multiple images having a similar exposure setting as the one exposure setting of the further image.

14. The method according to claim 12, wherein the masked image includes a boundary between the region including the object and the region excluding the object.

15. The method according to claim 14, wherein a transition of the boundary includes a binary transition or a sloping transition.

16. The method according to claim 14, wherein the step of merging the first and second radiance images includes applying a gradient weighting function to each of the masked first and second radiance images at the boundary.

17. The method according to claim 1, wherein the HDR image is formed without image registration.

18. The method according to claim 1, wherein the HDR image is formed without motion compensation.

19. An image processor for reconstructing a high dynamic range (HDR) image, comprising:

a controller that receives;

a) a first radiance image formed from multiple images of a scene captured over a range of different exposure settings and

b) a second radiance image formed from a further image including the scene and an object placed in the scene, the further image captured at one exposure setting;

a mask generator that generates a masked image of the object by comparing at least one of the multiple images to the further image; and

a merge generator that merges the first radiance image and the second radiance image using the masked image to reconstruct the HDR image, wherein the HDR image has a first region that includes the object and is formed from the second radiance image, wherein the HDR image has a second region that excludes the object and is formed from the first radiance image, and wherein the first region of the HDR image has a lower dynamic range than the second region of the HDR image.

20. The image processor according to claim 19, further comprising a radiance image generator that generates a) the first radiance image from the multiple images and b) the second radiance image from the further image based on a pixel intensity and the exposure settings in the respective multiple images and the further image.

21. The image processor according to claim 19, wherein the mask generator:

subtracts one of the multiple images from the further image to form the masked image, and forms an inverse masked image from an inverse of the masked image.

22. The image processor according to claim 21, wherein the mask generator applies the inverse masked image to the first radiance image and applies the masked image to the second radiance image, and the merge generator merges the masked first radiance image and the masked second radiance image received from the mask generator.

23. The image processor according to claim 21, wherein the controller applies the inverse masked image to the first radiance image and applies the masked image to the second radiance image, and the merge generator merges the masked first radiance image and the masked second radiance image received from the controller.

24. The image processor according to claim 21, wherein the merge generator applies the inverse masked image to the first radiance image and applies the masked image to the second radiance image, and merges the masked first radiance image and the masked second radiance image.

25. The image processor according to claim 21, wherein the masked image includes a boundary between the region including the object and the region excluding the object and the mask generator selects a sharp transition or a soft transition for the boundary.

26. The image processor according to claim 19, wherein the mask generator selects one of the multiple images having a similar exposure setting as the one exposure setting of the further image.

27. The image processor according to claim 19, wherein the scene is slowly varying with time and the range of different exposure settings represents a high dynamic range.

28. The image processor according to claim 27, wherein the object varies faster than the scene and the one exposure setting represents a low dynamic range.

29. The image processor according to claim 19, wherein the image processor reconstructs the HDR image without image registration.

30. The image processor according to claim 19, wherein the image processor reconstructs the HDR image without motion detection.

31. An imaging device for generating a high dynamic range (HDR) image comprising:

an image sensor, including a plurality of pixels, that captures an image; and

an image processor including;

a controller that receives;

a) a first radiance image formed from multiple images of a scene captured by the image sensor over a high dynamic range and

b) a second radiance image formed from a further image including the scene and an object placed in the scene, the further image captured by the image sensor with a low dynamic range, and a merge generator that merges the first radiance image and the second radiance image to generate the HDR image, wherein the merge generator merges portions of the second radiance image including the object with portions of the first radiance image substantially excluding the object, wherein the HDR image has a first region that includes the object and is formed from the second radiance image, wherein the HDR image has a second region that excludes the object and is formed from the first radiance image, and wherein the first region of the HDR image has a lower dynamic range than the second region of the HDR image.

32. The imaging device according to claim 31, further comprising a memory that stores one or more of the first radiance image, the second radiance image, the HDR image, the multiple images and the further image.

33. The imaging device according to claim 31, further comprising a mask generator that generates a masked image of the object by comparing at least one of the multiple images to the further image, wherein the merge generator merges the portions of the second radiance image with the portions of the first radiance image based on the masked image.

34. The imaging device according to claim 31, further comprising a radiance generator that generates the first and second radiance images based on a pixel intensity and the exposure settings of the respective multiple images and the further image.

35. The imaging device according to claim 31, further comprising:

a shutter that selectively passes light associated with the image;

imaging optics that focus the selectively passed light from the shutter to the image sensor; and

a display that displays the HDR image.

36. The imaging device according to claim 35, further comprising a user interface coupled to the image processor and that causes the imaging device to selectively capture the multiple images with the high dynamic range or the further image with the low dynamic range.

37. The imaging device according to claim 36, wherein at least one of the shutter, the imaging optics or the image sensor is adjusted based on the selective capture of the multiple images with the high dynamic range.

38. The imaging device according to claim 36, wherein at least one of the shutter, the imaging optics or the image sensor is adjusted based on selective capture of the further image with the low dynamic range.

39. The imaging device according to claim 31, wherein the high dynamic range is based on a corresponding range of exposure times for the multiple images.

40. The imaging device according to claim 31, wherein the low dynamic range is based on a predetermined exposure time to capture the object in the further image.