Systems and methods for highlight recovery in an image signal processor

• US 9,014,504 B2
• Filed: 05/31/2012
• Issued: 04/21/2015
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Lim, Suk Hwan, Hayward, David, Silverstein, D. Amnon
• Primary Examiner(s): OSINSKI, MICHAEL S
• Application Number:
  US13/485,199
• Publication Number:
  US 20130321679A1
• Time in Patent Office: 1,055 Days
• Field of Search: 3482071-20711, 348/222.1, 348/223.1, 348/225.1, 348/229.1, 34823199-2319, 348/239, 348/241, 348254-256, 348266-283, 348294-324, 34833301-33312, 348671-679, 382162-167, 382254-275
• US Class Current: 382/274
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 23/631 Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/136 using complementary colours
• CPC Class Codes: H04N 25/611 Correction of chromatic aberration

Abstract

Image sensors have finite ranges of illuminance that may be captured. When the sensors for particular pixels receive an amount of light exceeding these finite ranges, the pixel values clip to the maximum pixel value. Systems and methods for estimating pixel values that are clipped or near clipping are provided. In one example, a method for processing image data includes determining that a first channel of the image data is saturated or near saturation. The method further includes computing a highlight recovery value for the first channel based upon alternative channels in the image data that are not saturated or near saturation. The highlight recovery value is applied to the first channel.

Claims

1. A method for processing image data comprising:

determining that a first pixel of the image data is clipped or near clipping;

computing a highlight recovery value for the first pixel based upon alternative pixels in the image data that are not clipped or near clipping by;

querying at least one color lookup table for color samples, the color samples comprising pre-determined reference values stored in the color lookup tables;

linearly interpolating the highlight recovery values from the color samples provided by the color lookup tables; and

applying the highlight recovery value to the first pixel by replacing the pixel value data for the clipped pixels with the highlight recovery values.

2. The method of claim 1, comprising:

performing a simple demosaic process on the image data to obtain missing color samples for the image data.

3. The method of claim 2, wherein the image data comprises a red pixel, a blue pixel, a green-red pixel, and a green-blue pixel;

and

wherein performing the simple demosaic process on the image data produces;

interpolated data for the blue pixel, the green-red pixel and the green-blue pixel when a sample of the red pixel is present;

interpolated data for the red pixel, the green-red pixel and the green-blue pixel when a sample of the blue pixel is present;

interpolated data for the blue pixel, the red pixel and the green-blue pixel when a sample of the green-red pixel is present; and

interpolated data for the blue pixel, the green-red pixel and the red pixel when a sample of the green-blue pixel is present.

4. The method of claim 3, wherein the green-blue pixel and the green-red pixel are merged to form a green pixel by:

averaging interpolated green-red and green-blue pixel values for the red and blue pixels;

using a green-red value for the green-red pixel; and

using a green-blue value for the green-blue pixel.

5. The method of claim 1, wherein computing the highlight recovery value comprises:

computing a clip level for each pixel in the frame data; and

normalizing color values of each pixel based upon the computed clip level.

6. The method of claim 5, wherein computing the clip level comprises:

obtaining a saturation level and white balance gain for a sensor providing the image data;

obtaining a lens shading correction gain; and

obtaining the clip level by multiplying the saturation level by the lens shading correction gain.

7. The method of claim 6, wherein computing the clip level comprises:

obtaining a programmable offset and adding the programmable offset to the clip level.

8. The method of claim 5, wherein the normalized color values are represented by 17-bit unsigned values with 14 fractional bits.

9. The method of claim 5, wherein computing a highlight recovery value comprises:

using the normalized color value as the highlight recovery value when the normalized color value is less than a minimum register value associated with a color of the pixel that the highlight recovery value is being calculated for; and

when the normalized color value is greater than or equal to the minimum register value;

obtaining color sample values from a color lookup table associated with the color of the pixel that the highlight recovery value is being calculated for using the normalized color values as indices into the color lookup table;

interpolating a final output value from the color samples using linear interpolation; and

using the final output value as the highlight recovery value.

10. The method of claim 1, comprising:

applying post-processing to the highlight recovery value to limit the highlight recovery value to a maximum threshold value of all color pixel values.

11. An image signal processing system comprising:

circuitry, comprising;

highlight recovery logic configured to estimate and apply highlight recovery value data to image data for pixels that are clipped or near clipping, wherein clipped pixels comprise pixels having a pixel value at or above a sensor'"'"'s finite range of illuminance, wherein the highlight recovery value data is based at least in part upon pixel value data for pixels that are not clipped or near clipping; and

a plurality of color lookup tables associated with color pixels of the image data;

wherein the highlight recovery logic is configured to calculate and apply highlight recovery values to the clipped pixels in the image data by;

querying the color lookup tables for color samples, the color samples comprising pre-determined reference values stored in the color lookup tables;

linearly interpolating the highlight recovery values from the color samples provided by the color lookup tables; and

replacing the pixel value data for the clipped pixels with the highlight recovery values.

12. The image signal processing system of claim 11, wherein the estimated and applied highlight recovery value data comprises either a gain to be added to a current pixel value of the clipped pixels or a pixel value to replace the current pixel value of the clipped pixels.

13. The image signal processing system of claim 11, wherein the color samples in the color lookup tables are based upon preferred white balance settings, a particular time of day, or a subject of the raw image data.

14. The image signal processing system of claim 11, wherein the plurality of color lookup tables associated with color pixels of the raw image data comprise 4d color lookup tables;

wherein the 4d color lookup tables are indexed by;

normalized red values, normalized green values, normalized blue values, and a threshold value;

or

four normalized color pixel values.

15. The image signal processing system of claim 14, wherein the 4d color lookup tables are indexed by:

normalized red, green-red, green-blue, and blue values;

or

normalized red, green, blue, and white values.

16. The image signal processing system of claim 11, wherein the plurality of color lookup tables associated with color pixels of the raw image data comprise 5d color lookup tables;

wherein the 5d color lookup tables are indexed by normalized red values, normalized green values, normalized blue values, x-coordinates, and y-coordinates.

17. The image signal processing system of claim 11, wherein the plurality of color lookup tables associated with color pixels of the raw image data comprise 6d color lookup tables;

wherein the 6d color lookup tables are indexed by red, green, and blue values, and red, green, and blue clip values.

18. The image signal processing system of claim 11, wherein the plurality of color lookup tables associated with color pixels of the raw image data comprise 3d color lookup tables indexed by normalized red, green, and blue values.

19. The image signal processing system of claim 18, wherein the plurality of 3d color lookup tables comprise color sample values based upon a white balance, a current time of day, or a subject being captured by the image signal processing system.

20. The image signal processing system of claim 13, wherein the highlight recovery logic is configured to interpolate the highlight recovery values by linearly interpolating the recovery values from the queried lookup tables.

21. The image signal processing system of claim 11, comprising simple demosaic logic that is configured to:

interpolate red pixel values for green-blue, green-red, and blue pixels;

interpolate blue pixel values for green-blue, green-red, and red pixel values;

interpolate green-red pixel values for green-blue, blue, and red pixels; and

interpolate green-blue pixel values for green-red, red, and blue pixels.

22. The image signal processing system of claim 11, wherein the highlight recovery logic is configured to apply the highlight recovery value data to raw image data for clipped pixels based on position of the clipped pixels, color pixel of the clipped pixels, pixel values of pixels around the clipped pixels, one or more readings of another sensor, or a combination thereof.

23. The image signal processing system of claim 11, comprising:

lens shading correction logic configured to correct for distortions resulting from an intensity drop that increase with distance from an optical center of a lens;

white balance gain logic that is configured to produce a second output by taking a first output from the lens shading correction logic and adjusting a white balance of the first output;

wherein the highlight recovery logic is configured to estimate and apply the highlight recovery value data to the image data for the clipped pixels at or above the sensor'"'"'s finite range of illuminance after the lens shading correction logic and the white balance gain logic has been applied.

24. The image signal processing system of claim 11, comprising post processing logic configured to limit gains applied to a pixel to a programmable maximum pixel value.

25. One or more non-transitory tangible computer-readable storage media having instructions encoded thereon for execution by a processor, the instructions comprising:

code to determine a clipped pixel;

code to determine a highlight recovery value to extend a pixel value of the clipped pixel beyond a saturation value of a sensor associated with the clipped pixel based at least in part upon pixel values of non-clipped pixels by;

querying one or more color lookup tables for color samples, the color samples comprising pre-determined reference values stored in the color lookup tables;

linearly interpolating the highlight recovery value from the color samples provided by the color lookup tables; and

code to apply the highlight recovery value to the pixel value of the clipped pixel by replacing the pixel value data for the clipped pixels with the highlight recovery values.

26. The one or more non-transitory tangible computer-readable storage media of claim 25, wherein the code to determine the clipped pixel is configured to determine when a pixel intensity value is likely above a maximum level of intensity supported by an image sensor capturing an image.

27. The one or more non-transitory tangible computer-readable storage media of claim 26, comprising code to determine a clip level value, wherein the code to determine the clip level value is configured to multiply lens shading correction gains from lens shading correction logic to the maximum level of intensity supported by the image sensor and add a clip level offset associated with a color of the clipped pixel.

28. The one or more non-transitory tangible computer-readable storage media of claim 27, comprising code to normalize the clip level value, wherein the code to normalize the clip level value is configured to divide the pixel value by the clip level value associated with the color of the clipped pixel.

29. The one or more non-transitory tangible computer-readable storage media of claim 25, comprising code to bypass the code to determine the clipped pixel, the code to determine the highlight recovery value, and the code to apply the highlight recovery value to the pixel value of the clipped pixel when a bypass parameter is set.

30. The one or more non-transitory tangible computer-readable storage media of claim 25, wherein the code to determine the highlight recovery value is configured to:

obtain output values by querying a 3d lookup table associated with the color of the clipped pixel by using red, blue, and green pixel values for the clipped pixel as indices; and

linearly interpolate the output values to obtain the highlight recovery value.

31. An electronic device, comprising:

at least one digital image sensor, wherein the at least one digital image sensor has a finite range of illuminance;

an interface configured to communicate with the at least one digital image sensor;

a memory device;

a display device configured to display a visual representation of one or more image scenes corresponding to raw image data acquired by the at least one digital image sensor; and

an imaging signal processing sub-system comprising;

highlight recovery logic configured to extend pixel values for clipped pixels beyond the finite range of illuminance of the at least one digital image sensor by estimating a highlight recovery value based at least in part non-clipped pixels obtained by the at least one digital image sensor by;

querying one or more color lookup tables for color samples, the color samples comprising pre-determined reference values stored in the color lookup tables;

linearly interpolating the highlight recovery value from the color samples provided by the color lookup tables; and

applying the highlight recovery value to the pixel value of the clipped pixel by replacing the pixel value data for the clipped pixels with the highlight recovery values.

32. The electronic device of claim 31, wherein the highlight recovery logic is configured to determine the highlight recovery value by:

performing a simple demosaic to obtain interpolated color samples for both the pixels;

calculating a clip level based upon a maximum value defined by the finite range of illuminance, multiplying the maximum value by a lens shading correction gain provided by the imaging signal processing sub-system, and adding a clip level offset;

calculating a pixel intensity normalization by dividing the pixel values by the clip level associated with the color of the clipped pixel;

obtaining 3d color lookup table values based upon the pixel intensity normalization; and

interpolating the highlight recovery value using linear interpolation of the 3d color lookup table values.

33. The electronic device of claim 32, wherein the highlight recovery logic is configured to post-process the data to ensure that the highlight recovery value does not exceed a defined maximum threshold for the clipped pixel value.