Capturing and rendering high dynamic range images

• US 8,760,537 B2
• Filed: 09/03/2010
• Issued: 06/24/2014
• Est. Priority Date: 07/05/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Johnson, Garrett M., Cote, Guy, Orr, James Edmund IV
• Primary Examiner(s): VU, NGOC YEN T
• Application Number:
  US12/876,095
• Publication Number:
  US 20120002082A1
• Time in Patent Office: 1,390 Days
• Field of Search: 348/222.1, 348/229.1
• US Class Current: 348/229.1
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

Some embodiments of the invention provide a mobile device that captures and produces images with high dynamic ranges. To capture and produce a high dynamic range image, the mobile device of some embodiments includes novel image capture and processing modules. In some embodiments, the mobile device produces a high dynamic range (HDR) image by (1) having its image capture module rapidly capture a succession of images at different image exposure durations, and (2) having its image processing module composite these images to produce the HDR image.

Claims

1. A mobile device that captures images, the device comprising:

a camera for capturing at least three images at different image exposure durations; and

an image processing module for compositing the three captured images to generate a first composite image and modifying the first composite image to produce a second composite image, wherein modifying the first composite image to produce the second composite image comprises increasing a range of pixel values in a first set of regions of the first composite image and decreasing a range of pixel values in a second set of regions of the first composite image.

2. The mobile device of claim 1, wherein the three captured images are images of high dynamic range (HDR) scene.

3. The mobile device of claim 2, wherein the HDR scene has bright and dark regions and the first composite image is an image that displays details within the bright and dark regions of the HDR scene.

4. The mobile device of claim 1, wherein the image processing module compositing the three captured images comprises the image processing module identifying weighting values for pixels in the three captured images and selectively combining component color values of the pixels based on the identified weighting values.

5. The mobile device of claim 4, wherein the image processing module identifies the weighting values for the pixels based on brightness of the pixels.

6. The mobile device of claim 4, wherein the image processing module identifying the weighting values for the pixels comprises the image processing module identifying, for each pixel in each image a weighting value based on brightness values of neighboring pixels that are within a particular range of the pixel.

7. The mobile device of claim 1, wherein the image processing module is further for receiving the three captured images in a color space that is defined by luma and chroma channels, wherein the image processing module compositing the three images comprises the image processing module (i) selectively weighting and combining luma values of pixels in the three images to produce composite luma channel data and (ii) selectively weighting and combining chroma values of pixels in the three images to produce composite chroma channel data.

8. The mobile device of claim 7, wherein the image processing module is further for scaling the luma and chroma values to a desired range of values while compositing the three captured images.

9. The mobile device of claim 1, wherein the three captured images comprise luma channel data, wherein the image processing module is further for performing a linearization operation that converts the luma channel data to luminance channel data.

10. The mobile device of claim 9, wherein the image processing module performs the linearization operation before the image processing module composites the three images.

11. The mobile device of claim 9, wherein the image processing module performs the linearization operation while the image processing module composites the three images.

12. The mobile device of claim 1, wherein the three captured images are captured in an interval after the mobile device is directed to take a picture of a scene.

13. The mobile device of claim 1, wherein the image processing module comprises:

an alignment module for aligning the three captured images; and

a compositing module for the compositing of the three captured images after the three images are aligned.

14. The mobile device of claim 1, wherein the camera comprises:

a camera sensor for detecting light from a scene; and

an image capture module for interfacing with the camera sensor to capture the three images.

15. The mobile device of claim 1, wherein the mobile device is a digital camera.

16. The mobile device of claim 1, wherein the mobile device is one of a smartphone, a tablet computer, and a laptop computer.

17. The mobile device of claim 1, wherein the second composite image is a shadow-boosted composite image, wherein the image processing module is further for producing a highlight-attenuated third composite image, producing comprising the image processing module decreasing the range of pixel values in the first set of regions of the first composite image and increasing the range of pixel values in the second set of regions of the first composite image, wherein brightness values of the pixels in the second set of regions are greater than the brightness values of the pixels in the first set of regions.

18. The mobile device of claim 17, wherein the image processing module is further for compositing the shadow-boosted second composite image and the highlight-attenuated third composite image to generate a fourth composite image.

19. A non-transitory machine readable medium of a device that captures images, the medium storing a program for producing one high dynamic range (HDR) image, the program comprising sets of instructions for:

capturing a first image at a first exposure duration, a second image at a second exposure duration that is longer than the first exposure duration, and a third image at a third exposure duration that is longer than the second exposure duration;

generating, for each image, a mask that specifies parts of the image to use to produce the HDR image, wherein the mask for the second image is derived from the first image and the third image; and

using the masks to composite the captured images to produce the HDR image.

20. The machine readable medium of claim 19, wherein each mask comprises a plurality of mask values that specify the parts of the image to use to produce the HDR image, wherein the set of instructions for using the masks to composite the images comprises a set of instructions for using the mask values as weighting factors to selectively combine component color values of pixels in the images.

21. The machine readable medium of claim 20, wherein the mask values for the first image are based on brightness of the pixels in the first image and the mask values for the third image are based on brightness of the pixels in the third image.

22. The machine readable medium of claim 20, wherein the plurality of mask values of each image comprises a mask value for each pixel of the image, wherein the mask value for each particular pixel is based on brightness of pixels that neighbor the particular pixel.

23. The machine readable medium of claim 22, wherein the program further comprises a set of instructions for adjusting the mask value of each pixel in each image to improve contrast at boundaries between dark and bright regions of the image.

24. The machine readable medium of claim 19, wherein the set of instructions for generating the mask of each image comprises a set of instructions for quantifying a degree to which each specified part of the image is used in the composite.

25. The machine readable medium of claim 19, wherein the program further comprises a set of instructions for aligning the captured images before compositing the images.

26. The machine readable medium of claim 19, wherein the device is one of a smartphone, a camera, a laptop computer, and a desktop computer.

27. A method of generating a high dynamic range (HDR) image, the method comprising:

receiving a first image captured at a first exposure duration, a second image captured at a second exposure duration that is longer than the first exposure duration, and a third image captured at a third exposure duration that is longer than the second exposure duration;

for each image, using the second image to generate a mask that contains a masking value for each pixel in a plurality of pixels of the image;

for each image, adjusting a component color value of each pixel in the plurality of pixels of the image based on the pixel'"'"'s corresponding mask value and a factor related to the exposure duration at which the image was captured; and

generating a composite HDR image, wherein generating the composite HDR image comprises combining the adjusted component color values of corresponding pixels from each of the images.

28. The method of claim 27, wherein the adjusted component color value of each pixel is a luma value.

29. The method of claim 27, wherein the adjusted component color value of each pixel is a luminance value.

30. The method of claim 27, wherein the adjusted component color value of each pixel is a chroma value.

31. The method of claim 27, wherein using the second image to generate the mask comprises

using brightness values of pixels of the second image.

32. The method of claim 27, wherein using the second image to generate the mask for each image comprises using brightness values of the pixels in the second image for generating the masks for the first and third images.

33. The method of claim 32, wherein using the second image to generate the mask for each image further comprises using brightness values of the pixels in the second image to generate the mask for the second image.

34. The method of claim 27 further comprising blurring each of the masks to make the masking values spatially smooth in order to reduce high frequency transitions between the exposures.

35. The method of claim 34, wherein reducing the high frequency transitions between the exposures reduces a loss of contrast in the composite HDR image at boundaries between dark and bright regions in the image.

36. The method of claim 27, wherein generating the composite HDR image comprises:

defining an initial composite image from the combined adjusted component color values;

generating a blurred image from the initial composite image, generating the blurred image comprising adjusting the combined component color values;

generating a shadow-enhanced version of the initial composite image based on the initial composite image and the blurred image;

generating a highlight-attenuated version of the initial composite image based on the initial composite image and the blurred image; and

generating the composite HDR image based on the shadow-enhanced and the highlight-attenuated versions of the initial composite image.

37. The method of claim 27, wherein the adjusted component color values are luma values or luminance values.

38. The method of claim 27, wherein the received images are images of an HDR scene, wherein the composite image is an image that displays details within the HDR scene.