Systems and methods for local tone mapping

• US 9,105,078 B2
• Filed: 05/31/2012
• Issued: 08/11/2015
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Lim, Suk Hwan, Silverstein, D. Amnon, Cote, Guy, Hordley, Steven David, Ku, Weichun, Bratt, Joseph P., Finlayson, Graham
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US13/485,421
• Publication Number:
  US 20130322753A1
• Time in Patent Office: 1,167 Days
• Field of Search: 382/162, 382/167, 382/260, 382/263, 382/264, 345/102, 345/204, 345/214, 345/589, 345/690, 358/1.9, 358/3.01, 348/272, 348/597, 348/231.3
• US Class Current: 1/1
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 2207/20028 Bilateral filtering
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement

Abstract

Systems and methods for local tone mapping are provided. In one example, an electronic device includes an electronic display, an imaging device, and an image signal processor. The electronic display may display images of a first bit depth, and the imaging device may include an image sensor that obtains image data of a higher bit depth than the first bit depth. The image signal processor may process the image data, and may include local tone mapping logic that may apply a spatially varying local tone curve to a pixel of the image data to preserve local contrast when displayed on the display. The local tone mapping logic may smooth the local tone curve applied to the intensity difference between the pixel and another nearby pixel exceeds a threshold.

Claims

1. An image signal processing system comprising:

local tone mapping logic to apply spatially varying tone curves to image data, wherein the local tone mapping logic comprises a spatially varying grid of lookup tables of tone curves indexed by luminance associated each pixel of the image data; and

spatially varying color correction matrices to apply spatially varying color correction matrices to the image data, wherein each of the spatially varying color matrices are determined based on each spatial location of each pixel with respect to a frame of the image data.

2. The image signal processing system of claim 1, wherein each lookup table of the spatially varying grid of lookup tables of tone curves is indexed by:

an average luminance of all color components of a pixel of the image data;

a maximal luminance of the color components of the pixel;

a blend of the average luminance and the maximal luminance;

or

a logarithmic value of the average luminance, the maximal luminance, or the blend of the average luminance and the maximal luminance;

or

a combination thereof.

3. The image signal processing system of claim 1, wherein the local tone mapping logic comprises luminance computation logic configured to determine the luminance associated with each pixel of the image data.

4. The image signal processing system of claim 1, wherein the spatially varying color correction matrices are stored in a spatially varying grid of lookup tables indexed by luminance associated each pixel of the image data.

5. The image signal processing system of claim 1, comprising white pin logic configured to blend white into substantially saturated image data when the substantially saturated image data would otherwise appear gray.

6. An image signal processing system comprising:

direct memory access logic to receive a frame of RGB-format image data into logic of an image signal processor;

local tone mapping logic to apply spatially varying tone curves to a pixel of the image data, wherein the local tone mapping logic comprises a spatially varying grid of lookup tables of tone curves indexed by luminance associated each pixel of the image data; and

spatially varying color correction matrix logic to apply a color correction matrix to the pixel of the image that varies depending on;

the spatial location of the pixel within the image frame; and

a luminance value associated with the pixel or a value of one of the color components of the pixel, or a combination thereof.

7. The image signal processing system of claim 6, wherein the luminance value comprises:

an average luminance of all color components of the pixel;

a maximal luminance of the color components of the pixel;

a blend of the average luminance and the maximal luminance;

or

a logarithmic value of the average luminance, the maximal luminance, or the blend of the average luminance and the maximal luminance;

or

a combination thereof.

8. A method for local tone mapping in an image signal processor, comprising:

receiving, via a processor, a pixel into luminance computation logic of the image signal processor to compute an input luminance associated with the pixel;

obtaining, via the processor, an output luminance from a spatially varying lookup table of local tone curves indexed to the input luminance, wherein each entry of the spatially varying lookup table of local tone curves is associated with a spatial location of the pixel within a frame of image data;

computing, via the processor, a pixel gain based at least in part on the input luminance and the output luminance;

applying, via the processor, spatially varying tone curves to the pixel based on the spatially varying lookup table of local tone curves to generated a tone corrected pixel;

applying, via the processor, a spatially varying color correction matrix to the tone corrected pixel to obtain a color corrected pixel, wherein the spatially varying color correction matrix is associated with a spatial location of the pixel within the frame of the image data; and

applying, via the processor, the pixel gain to the color corrected pixel to obtain a locally tone-mapped color-corrected pixel.

9. The method of claim 8, wherein the application of the spatially varying color correction matrix varies depending on the input luminance of the pixel.

10. The method of claim 8, comprising converting the input luminance into log space to provide higher resolution for dark colors, wherein the output luminance of the spatially varying lookup table is provided in log space, and comprising converting the output luminance into linear space before computing the pixel gain.

11. The method of claim 8, comprising blending white into the locally tone-mapped color-corrected pixel when the pixel is substantially saturated but would otherwise appear substantially gray.