Adaptive lens shading correction

• US 8,228,406 B2
• Filed: 06/04/2010
• Issued: 07/24/2012
• Est. Priority Date: 06/04/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Kuo, David Daming, Guo, Haitao, Zipnick, Jay David, Chen, Ting, Chawla, Sumit
• Primary Examiner(s): Chan, Jason
• Assistant Examiner(s): Vieaux, Gary C
• Application Number:
  US12/794,632
• Publication Number:
  US 20110298944A1
• Time in Patent Office: 781 Days
• Field of Search: 348/222.1, 348/251, 382/167, 382/274
• US Class Current: 348/251
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
• CPC Class Codes: H04N 25/611 Correction of chromatic aberration

Abstract

Systems, methods, and devices for applying lens shading correction to image data captured by an image sensor are provided. In one embodiment, multiple lens shading adaptation functions, each modeled based on the response of a color channel to a reference illuminant, are provided. An image frame from the image data may be analyzed to select a lens shading adaptation function corresponding to a reference illuminant that most closely matches a current illuminant. The selected lens shading function may then be used to adjust a set of lens shading parameters.

Claims

1. A method comprising:

providing a set of lens shading adaptation functions, wherein each of the lens shading adaptation functions corresponds to a respective reference illuminant;

providing a plurality of image frames captured by an image sensor under current illuminant conditions to a hardware image signal processing pipeline;

using an additional image processing pipeline to perform alternative image processing by analyzing a reference frame from a plurality of image frames to select the lens shading adaptation function corresponding to a reference illuminant that most closely matches characteristics of the current illuminant conditions;

adjusting a set of lens shading parameters based upon the selected lens shading adaptation function;

providing the adjusted lens shading parameters to a hardware image signal processing pipeline as feed-forward parameters; and

using the hardware image signal processing pipeline to process the plurality of image frames based on the adjusted lens shading parameters.

2. The method of claim 1, wherein the plurality of image frames captured by the image sensor comprises red, green, and blue color channels.

3. The method of claim 2, wherein analyzing the reference frame to select the lens shading adaptation function comprises:

identifying generally neutral regions within the reference frame;

applying each of the lens shading adaptation functions to the reference frame and determining, for each of the lens shading adaptation functions, a variance of red to blue ratio values within the neutral regions; and

selecting the lens shading adaptation function corresponding to a minimum variance if the minimum variance exists.

4. The method of claim 3, wherein, if a minimum variance does not exist, a lens shading adaptation function is not selected, the lens shading parameters are not adjusted, and the hardware image signal processing pipeline processes the plurality of image frames using the unadjusted lens shading parameters.

5. The method of claim 3, wherein identifying generally neutral regions within the reference frame comprises:

analyzing subsets of the reference frame;

for each subset, determining averaged color values for each of the blue and green color channels and calculating a blue to green ratio based upon the averaged blue color value and the averaged green color value; and

identifying the subsets within the reference frame having similar blue to green ratios as generally neutral regions.

6. The method of claim 5, wherein each subset comprises an 8×

8 block of pixels.

7. The method of claim 6, wherein the 8×

8 block of pixels comprises Bayer image data having a first green channel and a second green channel, and wherein the averaged green color value is determined based upon the first green channel, the second green channel, or a combination thereof.

8. A system comprising:

a first image processing pipeline configured to receive frames of image data generated by an image sensor and to process the frames of image data to compensate for lens shading fall-off in the frames using a set of lens shading parameters;

a feed-forward loop comprising a second image processing pipeline configured to receive, in parallel with the first image processing pipeline, at least one of the frames of image data generated by the image sensor and to select a lens shading adaptation function from a set of available lens shading adaptation functions to compensate for the sensitivity of a lens shading response due to certain wavelengths of light for at least one color channel of the image data;

wherein the feed-forward loop is configured to adjust the set of lens shading parameters used by the first image processing pipeline based on the selected lens shading adaptation function.

9. The system of claim 8, wherein the at least one color channel comprises a red color channel.

10. The system of claim 9, wherein the certain wavelengths comprise light having a wavelength of between approximately 600 to 650 nanometers.

11. The system of claim 8, wherein each of the set of available lens shading adaptation functions is modeled based upon the lens shading response of the at least one color channel to a respective one of a set of reference illuminants.

12. The system of claim 11, wherein the set of reference illuminants comprises at least one of a simulated daylight illuminant, a fluorescent illuminant, and an incandescent illuminant.

13. The system of claim 8, wherein the second image processing pipeline comprises a memory encoded with image analysis software executable by a processor, and wherein the second image processing pipeline is configured to analyze the at least one frame by execution of the image analysis software by the processor.

14. A method comprising:

using a hardware image processing pipeline to receive a plurality of image frames generated by a digital image sensor and to apply lens shading correction to each of the plurality of image frames on a per-pixel basis based upon a set of lens shading gains.

using a second image processing pipeline to store at least one image frame from the plurality of image frames and to analyze the at least one image frame, wherein the second image processing pipeline is configured to select a set of lens shading adaptation values based upon the analysis, to adjust the set of lens shading gains based upon the selected set of lens shading adaptation values, and to provide the adjusted set of lens shading gains to the hardware image processing pipeline as feed-forward control parameters.

15. The method of claim 14, wherein the set of lens shading adaptation values is selected from a plurality of sets of lens shading adaptation values, wherein each of the sets of lens shading adaptation values is modeled based at least partially upon the response to different types of lighting conditions of an infrared cutoff filter associated with the digital image sensor.

16. The method of claim 14, wherein the second image processing pipeline is configured to store and analyze the at least one image frame upon detecting a change in the lighting conditions.

17. The method of claim 14, wherein detecting the change in the lighting conditions comprises detecting a change in auto-white balance values and waiting for the auto-white balance values to stabilize.

18. The method of claim 17, wherein waiting for the auto-white balance values to stabilize comprises determining if the auto-white balance values remain stable for at least two image frames.

19. An electronic device comprising:

a processor;

a memory containing image analysis software executable by the processor;

an image capture device comprising an image sensor configured to capture raw image data;

a first image processing pipeline configured to receive the raw image data and to apply lens shading correction to the raw image data using a set of lens shading parameters; and

a feed-forward loop comprising a second image processing pipeline configured to receive at least one frame of the raw image data, to analyze the at least one frame using the image analysis software to select a lens shading adaptation function for the at least one frame, and to adjust the lens shading parameters using one or more intermediate transition steps based upon the selected lens shading adaptation function over a plurality of image frames to obtain a final set of adjusted lens shading parameters;

wherein the first image processing pipeline is configured to receive the intermediate adjusted lens shading parameters and the final adjusted lens shading parameters as feed-forward outputs from the feed-forward loop and to apply the intermediate and adjusted lens shading parameters to the raw image data.

20. The electronic device of claim 19, comprising an infinite impulse response (IIR) filter configured to determine the one or more intermediate transition steps.

21. The electronic device of claim 20, wherein the adjustment of the lens shading parameters by the image analysis software using the one or more transition steps comprises:

determining a total difference between the currently selected lens shading adaptation function and a lens shading adaptation function selected for the previous frame analyzed by the second image processing pipeline;

during a first transition step, adjusting the lens shading adaptation values corresponding to the previously selected lens shading adaptation function towards the lens shading values corresponding to the currently selected lens shading adaptation function by approximately 50 percent of the total difference during a first set of one or more frames to produce a first intermediate set of lens shading adaptation values;

during a second transition step, adjusting the first intermediate set of lens shading adaptation values towards the lens shading values corresponding to the currently selected lens shading adaptation function by approximately 25 percent of the total difference during a second set of one or more frames subsequent to the first set of one or more frames to produce a second intermediate set of lens shading adaptation values;

during a third transition step, adjusting the second intermediate set of lens shading adaptation values towards the lens shading values corresponding to the currently selected lens shading adaptation function by approximately 12.5 percent of the total difference during a third set of one or more frames subsequent to the second set of one or more frames to produce a third intermediate set of lens shading adaptation values; and

during a fourth transition step, setting the lens shading adaptation values corresponding to the currently selected lens shading adaptation function as the adjusted lens shading parameters.

22. The electronic device of claim 21, wherein the image processing pipeline is configured to apply lens shading correction to the raw image data based upon the first intermediate set of lens shading adaptation values during the first set of one or more frames, apply lens shading correction to the raw image data based upon the second intermediate set of lens shading adaptation values during the second set of one or more frames, and apply lens shading correction to the raw image data based upon the third intermediate set of lens shading adaptation values during the third set of one or more frames.

23. The electronic device of claim 19, wherein the at least one frame is received via a memory using a direct memory access (DMA) interface.

24. The electronic device of claim 19, wherein the lens shading adaptation function is selected from a plurality of available lens shading adaptation functions, each being modeled as a fourth power polynomial based on the distance from an optical center of a lens of the image capture device corresponding to the lens shading response of a red color channel to different types of illuminants.

25. The electronic device of claim 19, wherein the image sensor comprises a Bayer image sensor.

26. The electronic device of claim 19, comprising a desktop computer, a laptop computer, a tablet computer, a mobile cellular telephone, a portable digital media player, or some combination thereof.