SYSTEM AND METHOD FOR APPLYING LENS SHADING CORRECTION DURING IMAGE PROCESSING

• US 20110091101A1
• Filed: 10/20/2009
• Published: 04/21/2011
• Est. Priority Date: 10/20/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Cote, Guy, Frederiksen, Jeffrey E.
• Granted Patent: US 8,472,712 B2
• Time in Patent Office: Days
• US Class Current: 382/167
• CPC Class Codes: G06T 5/008 Local, e.g. shadow enhancement
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: H04N 23/631 Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/136 using complementary colours
• CPC Class Codes: H04N 25/611 Correction of chromatic aberration
• CPC Class Codes: H04N 25/68 applied to defects
• CPC Class Codes: H04N 25/683 by defect estimation performed on the scene signal, e.g. real time or on the fly detection

Abstract

Various techniques for lens shading correction are provided. In one embodiment, the location of a current pixel is determined relative to a gain grid having a plurality of grid points distributed in horizontal and vertical directions. If the location of the current pixel corresponds to a grid point, a lens shading gain associated with that grid point is applied to the current pixel. If the location of the current pixel is between four grid points, bi-linear interpolation is applied to the four grid points to determine an interpolated lens shading gain. In another embodiment, a radial gain grid may be provided, and lens shading gains may be interpolated based upon grid points neighboring a current pixel in the radial and angular directions. In a further embodiment, a radial lens shading gain is determined by determining a radial distance from the center of the image to the current pixel and multiplying the radial distance by a global gain parameter based upon the color of the current pixel. The radial lens shading gain is then applied to the current pixel, along with the determined lens shading grid gain or lens shading interpolated gain.

Claims

1. A method for applying lens shading correction to image data, comprising:

using an image processor;

receiving an input pixel of the image data, wherein the image data is acquired using an image sensor;

determining a color component of the input pixel;

determining the location of the input pixel relative to a gain grid comprising a plurality of grid points distributed in horizontal and vertical directions, wherein each of the plurality of grid points is associated with a lens shading gain selected based upon the color of the input pixel;

if the location of the input pixel corresponds directly to one of the plurality of grid points, applying the lens shading gain associated with the directly corresponding grid point to the input pixel; and

if the location of the input pixel does not correspond directly to one of the plurality of grid points, identifying a nearest set of grid points that enclose the input pixel, determining a lens shading gain by interpolating the lens shading gains associated with each of the set of grid points, and applying the interpolated lens shading gain to the input pixel.

2. The method of claim 1, wherein a lens shading correction region is defined within an imaging frame of the image sensor and is enclosed by the boundaries of the gain grid, and wherein a lens shading gain is not applied to the input pixel if the location of the input pixel is outside of the lens shading correction region.

3. The method of claim 1, wherein the gain grid comprises a rectangular gain grid, and wherein the set of grid points that enclose the input pixel comprises a first grid point, a second grid point, a third grid point, and a fourth grid point, and wherein the locations of each of the first, second, third, and fourth grid points are to the top-left, top-right, bottom-left, and bottom-right, respectively, relative to the location of the input pixel.

4. The method of claim 3, wherein the interpolated lens shading gain is determined using bi-linear interpolation of the lens shading gains associated with the first, second, third, and fourth grid points in the horizontal and vertical directions.

5. The method of claim 4, wherein using bi-linear interpolation to determine the interpolated lens shading gain comprises:

multiplying the lens shading gain associated with the first grid point by the vertical distance between the input pixel and the third and fourth grid points and by the horizontal distance between the input pixel and the second and fourth grid points to obtain a first value;

multiplying the lens shading gain associated with the second grid point by the vertical distance between the input pixel and the third and fourth grid points and by the horizontal distance between the input pixel and the first and third grid points to obtain a second value;

multiplying the lens shading gain associated with the third grid point by the vertical distance between the input pixel and the first and second grid points and by the horizontal distance between the input pixel and the second and fourth grid points to obtain a third value;

multiplying the lens shading gain associated with the fourth grid point by the vertical distance between the input pixel and the first and second grid points and by the horizontal distance between the input pixel and the first and third grid points to obtain a fourth value;

summing the first, second, third, and fourth values to obtain a fifth value; and

dividing the fifth value by the product of the horizontal interval and the vertical interval to determine the interpolated lens shading gain for the input pixel.

6. The method of claim 1, wherein the gain grid comprises a radial gain grid, and wherein the interpolated lens shading gain is determined using bi-linear interpolation of the lens shading gains associated with grid points that enclose the input pixel in the radial and angular directions.

7. The method of claim 1, wherein, for each color component of the image sensor, the grid points are distributed at fixed horizontal and vertical intervals.

8. The method of claim 7, wherein the horizontal and vertical intervals between grid points vary depending on the color of a current input pixel.

9. The method of claim 1, wherein the grid points are distributed at non-fixed intervals or at logarithmic intervals.

10. A method for applying lens shading correction to image data, comprising:

using an image processor;

receiving an input pixel of the image data, wherein the image data is acquired using an image sensor;

determining the location of the input pixel within a lens shading correction region defined within an imaging frame of the image sensor relative to a gain grid comprising a plurality of grid points distributed in horizontal and vertical directions, wherein each of the plurality of grid points is associated with a lens shading gain;

determining an interpolated lens shading gain for the input pixel by interpolating the lens shading gains associated with a nearest set of grid points enclosing the input pixel in the horizontal and vertical directions;

identifying a pixel corresponding to the center of the lens shading correction region;

determining a radial distance between the center pixel and the input pixel;

determining a radial lens shading gain based at least partially upon the radial distance; and

applying the interpolated lens shading gain and the radial lens shading gain to the input pixel.

11. The method of claim 10, wherein the lens shading gains associated with each of the plurality of grid points are shared for all pixels of the image data regardless of color.

12. The method of claim 10, wherein determining the radial distance comprises:

squaring the difference between the horizontal coordinate of the input pixel and the horizontal coordinate of the center pixel to obtain a first value;

squaring the difference between the vertical coordinate of the input pixel and the vertical coordinate of the center pixel to obtain a second value;

summing the first and second values to obtain a third value; and

taking the square-root of the third value to determine the radial distance.

13. The method of claim 10, wherein determining the radial distance comprises approximating the radial distance by multiplying the greater of either the difference between the horizontal coordinate of the input pixel and the horizontal coordinate of the center pixel or the difference between the vertical coordinate of the input pixel and the vertical coordinate of the center pixel by a first estimation coefficient to obtain a first value, multiplying the lesser of either the difference between the horizontal coordinate of the input pixel and the horizontal coordinate of the center pixel or the difference between the vertical coordinate of the input pixel and the vertical coordinate of the center pixel by a second estimation coefficient to obtain a second value, and summing the first and second values to approximate the radial distance.

14. The method of claim 13, wherein the first estimation coefficient is between approximately 0.35 and 0.43, and wherein the second estimation coefficient is between approximately 0.92 and 0.98.

15. The method of claim 10, wherein determining a radial lens shading gain comprises:

determining the color of the input pixel;

selecting a global gain parameter based upon the color of the input pixel; and

multiplying the selected global gain parameter by the radial distance.

16. An image signal processing circuit comprising:

an interface configured to receive image data acquired using an image sensor of an imaging device; and

an image processing pipeline configured to perform a set of processing operations on the image data, wherein one of the set of processing operations is a lens shading correction operation, and wherein the image processing pipeline comprises lens shading correction logic configured to;

receive an input pixel of the image data;

determine the location of the input pixel within a lens shading correction region defined within an imaging frame of the image sensor relative to a gain grid having a plurality of grid points distributed horizontally and vertically, wherein each of the plurality of grid points is associated with a lens shading gain;

determine an interpolated lens shading gain for the input pixel using bi-linear interpolation of the lens shading gains associated with a nearest set of grid points enclosing the input pixel;

determine a radial lens shading gain as a function of a global gain parameter associated with the color of the input pixel and a radial distance between the center of the lens shading correction region and the location of the input pixel; and

apply the interpolated lens shading gain and the radial lens shading gain to the input pixel.

17. The image signal processing circuit of claim 16, wherein the interface comprises a Standard Mobile Imaging Architecture (SMIA) interface or a Mobile Industry Processor Interface (MIPI), or some combination thereof.

18. The image signal processing circuit of claim 16, comprising front-end processing logic configured to perform an initial set of operations on the image data before the image data is processed by the image processing pipeline, wherein the front-end processing logic is configured to receive an input pixel of the image data, determine the location of the input pixel within a lens shading correction region defined within an imaging frame of the image sensor relative to a gain grid having a plurality of grid points distributed horizontally and vertically, wherein each of the plurality of grid points is associated with a lens shading gain, determine an interpolated lens shading gain for the input pixel using bi-linear interpolation of the lens shading gains associated with a nearest set of grid points enclosing the input pixel, determine a radial lens shading gain as a function of a global gain parameter associated with the color of the input pixel and a radial distance between the center of the lens shading correction region and the location of the input pixel, apply the interpolated lens shading gain and the radial lens shading gain to the input pixel, and collect imaging statistics on the image data after the interpolated lens shading gain and the radial lens shading gain are applied.

19. The image signal processing circuit of claim 18, wherein the collected imaging statistics comprise white balance statistics, exposure statistics, or focusing statistics, or some combination thereof.

20. The image signal processing circuit of claim 18, comprising control logic configured to receive the collected imaging statistics and to determine one or more control parameters for the imaging device, or one or more control parameters for the image processing pipeline, or some combination thereof.

21. An electronic device, comprising:

an imaging device comprising an image sensor; and

image processing circuitry configured to process image data acquired using the image sensor, wherein the image processing circuitry comprises;

lens shading correction logic configured to;

determine the location of an input pixel of the image data within a lens shading correction region defined within an imaging frame of the image sensor relative to a gain grid having a plurality of grid points distributed horizontally and vertically, wherein each of the plurality of grid points is associated with a lens shading gain;

determine an interpolated lens shading gain for the input pixel using bi-linear interpolation of the lens shading gains associated with a nearest set of grid points enclosing the input pixel;

determine a radial lens shading gain as a function of a global gain parameter associated with the color of the input pixel and a radial distance between the center of the lens shading correction region and the location of the input pixel; and

apply the interpolated lens shading gain and the radial lens shading gain to the input pixel.

22. The electronic device of claim 21, wherein the image sensor comprises a color filter array, and wherein each pixel of the image data is associated with a color component based upon the configuration of the color filter array.

23. The electronic device of claim 22, wherein the color filter array comprises a Bayer filter, a RGBW filter, or a CYGM filter.

24. The electronic device of claim 22, comprising a memory device, wherein the lens shading gains associated with the plurality of grid points of the gain grid are stored in the memory device and are shared for each of color component of the color filter array.

25. The electronic device of claim 21, wherein the image sensor comprises a complementary metal-oxide image semiconductor (CMOS) image sensor, a charge-coupled device (CCD) image sensor, or some combination thereof.

26. The electronic device of claim 21, comprising a display device configured to display the image data subsequent to processing by the lens shading correction logic, and wherein the processed image data exhibits increased uniformity in light intensity.

27. The electronic device of claim 21, wherein the electronic device comprises a laptop computer, a desktop computer, or a handheld portable device.

28. One or more tangible computer-readable storage media having instructions encoded thereon for execution by a processor, the instructions comprising:

code to determine the location of an input pixel within a lens shading correction region defined within an imaging frame of an image sensor relative to a gain grid having a plurality of grid points distributed horizontally and vertically, wherein each of the plurality of grid points is associated with a lens shading gain;

code to identify a set of grid points nearest to the location of the input pixel, wherein the set of grid points encloses the input pixel;

code to compute a interpolated lens shading gain for the input pixel based upon the lens shading gains associated with the identified set of grid points;

code to compute a radial lens shading gain based upon a radial distance between the center of the lens shading correction region and the location of the input pixel; and

code to cause at least one of the interpolated lens shading gain or the radial lens shading gain to be applied to the input pixel.

29. The one or more tangible computer-readable storage media of claim 28, wherein the radial lens shading gain is additionally based upon a global gain parameter selected based upon the color of the input pixel.