Systems and methods for local tone mapping

• US 9,741,099 B2
• Filed: 08/10/2015
• Issued: 08/22/2017
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Lim, Suk Hwan, Silverstein, D. Amnon, Cote, Guy, Hordley, Steven David, Finlayson, Graham, Ku, Weichun, Bratt, Joseph P.
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US14/822,316
• Publication Number:
  US 20150348246A1
• Time in Patent Office: 743 Days
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 2207/20028 Bilateral filtering
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement

Abstract

Systems and methods for local tone mapping are provided. In one example, an electronic device includes an electronic display, an imaging device, and an image signal processor. The electronic display may display images of a first bit depth, and the imaging device may include an image sensor that obtains image data of a higher bit depth than the first bit depth. The image signal processor may process the image data, and may include local tone mapping logic that may apply a spatially varying local tone curve to a pixel of the image data to preserve local contrast when displayed on the display. The local tone mapping logic may smooth the local tone curve applied to the intensity difference between the pixel and another nearby pixel exceeds a threshold.

Claims

1. An electronic device comprising:

an electronic display configured to display images of a first bit depth;

an imaging device comprising an image sensor configured to obtain image data of a higher bit depth than the first bit depth; and

an image signal processor configured to process the image data, wherein the image signal processor comprises;

local tone mapping logic configured to apply a spatially varying local tone curve to a pixel of the image data to preserve local contrast when displayed on the display, wherein the local tone mapping logic is configured to smooth the local tone curve applied to the pixel unless an intensity difference between the pixel and one or more other nearby pixels exceeds a threshold indicative of an edge within the image data, wherein the local tone mapping logic is configured to apply the spatially varying local tone curve to the pixel by computing a gain based at least in part on an input luminance associated with the pixel and an output luminance obtained by a spatially varying lookup table.

2. The electronic device of claim 1, wherein the local tone mapping logic is configured to compute the gain as the output luminance divided by the input luminance.

3. The electronic device of claim 1, wherein the local tone mapping logic is configured to compute the gain as a smoothed value of the output luminance divided by the input luminance unless the intensity difference between the pixel and the one or more other nearby pixels exceeds the threshold.

4. The electronic device of claim 1, wherein the local tone mapping logic comprises a bilateral filter configured to vary the gain depending on whether the intensity difference between the pixel and the one or more other nearby pixels exceeds the threshold.

5. The electronic device of claim 4, wherein the bilateral filter comprises a horizontal bilateral filter.

18. The electronic device of claim 1, wherein the image signal processor comprises white pin logic configured to blend white into substantially saturated image data when the substantially saturated image data would otherwise appear gray.

6. An electronic device comprising:

an imaging device configured to obtain image data; and

an image signal processor configured to process the image data, wherein the image signal processor comprises;

local tone mapping logic configured to apply a spatially varying local tone curve to a pixel of the image data to preserve local contrast when displayed on the display, wherein the local tone mapping logic is configured to smooth the local tone curve applied to the pixel unless an intensity difference between the pixel and one or more other nearby pixels exceeds a threshold indicative of an edge within the image data, wherein the local tone mapping logic is configured to apply the spatially varying local tone curve to the pixel by computing a gain based at least in part on an input luminance associated with the pixel and an output luminance obtained by a spatially varying lookup table.

7. The electronic device of claim 6, wherein the image signal processor comprises white pin logic configured to blend white into substantially saturated image data when the substantially saturated image data would otherwise appear gray, wherein the white pin logic comprises compensation gain logic to apply a spatially varying compensation gain value to either a maximum or minimum color component value, or both, of a pixel of the image data to determine when the pixel of the image data is substantially saturated but would otherwise appear gray.

8. The electronic device of claim 7, wherein the compensation gain logic is configured to determine the spatially varying compensation gain value using a spatially varying compensation gain lookup table, wherein the spatially varying compensation gain lookup table comprises compensation gains that decrease radially from an optical center of the image data.

9. The electronic device of claim 8, wherein the spatially varying compensation gain lookup table is configured to be indexed by a maximum or minimum color component value of an input pixel.

10. The electronic device of claim 7, wherein the white pin logic is a component of an RGB-format image processing pipeline.

11. The electronic device of claim 6, wherein the imaging device comprises a digital camera integrated with the electronic device, an external digital camera coupled to the electronic device via an input/output port, or some combination thereof.

12. The electronic device of claim 6, comprising at least one of a desktop computer, a laptop computer, a tablet computer, a mobile cellular telephone, a portable media player, or any combination thereof.

13. An image signal processor configured to process image data acquired by an image sensor, wherein the image signal processor comprises:

means for implementing local tone mapping logic configured to;

apply a spatially varying local tone curve to a pixel of the image data to preserve local contrast when displayed on the display; and

smooth the local tone curve applied to the pixel unless an intensity difference between the pixel and one or more other nearby pixels exceeds a threshold indicative of an edge within the image data, wherein the local tone mapping logic is configured to apply the spatially varying local tone curve to the pixel by computing a gain based at least in part on an input luminance associated with the pixel and an output luminance obtained by a spatially varying lookup table.

14. The image signal processor of claim 13, comprising means for implementing white pin logic configured to blend white into substantially saturated image data when the substantially saturated image data would otherwise appear gray.

15. The image signal processor of claim 14, wherein the white pin logic comprises means for compensation gain logic to apply a spatially varying compensation gain value to either a maximum or minimum color component value, or both, of a pixel of the image data to determine when the pixel of the image data is substantially saturated but would otherwise appear gray.

16. The image signal processor of claim 15, wherein the compensation gain logic is configured to determine the spatially varying compensation gain value using a spatially varying compensation gain lookup table, wherein the spatially varying compensation gain lookup table comprises compensation gains that decrease radially from an optical center of the image data.

17. The image signal processor of claim 14, comprising means for highlight recovery logic configured to recover image information from clipped or nearly clipped pixels of the image data before reaching the white pin logic.