Systems and methods for defective pixel correction with neighboring pixels

• US 9,131,196 B2
• Filed: 12/21/2012
• Issued: 09/08/2015
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Lim, Suk Hwan, Mills, Christopher L., Lin, Sheng
• Primary Examiner(s): Henn, Timothy J
• Assistant Examiner(s): PHAM, QUAN L
• Application Number:
  US13/724,574
• Publication Number:
  US 20130329098A1
• Time in Patent Office: 991 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: H04N 23/631 Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/136 using complementary colours
• CPC Class Codes: H04N 25/611 Correction of chromatic aberration
• CPC Class Codes: H04N 25/67 applied to fixed-pattern noise, e.g. non-uniformity of response
• CPC Class Codes: H04N 25/68 applied to defects
• CPC Class Codes: H04N 5/142 Edging; Contouring
• CPC Class Codes: H04N 9/64 Circuits for processing colour signals H04N9/77 takes precedence; camera processing pipelines for processing colour signals H04N23/84

Abstract

The present disclosure generally relates to systems and methods for image data processing. In certain embodiments, an image processing pipeline may detect and correct a defective pixel of image data acquired using an image sensor. The image processing pipeline may receive an input pixel of the image data acquired using the image sensor. The image processing pipeline may then identify a set of neighboring pixels having the same color component as the input pixel and remove two neighboring pixels from the set of neighboring pixels thereby generating a modified set of neighboring pixels. Here, the two neighboring pixels correspond to a maximum pixel value and a minimum pixel value of the set of neighboring pixels. The image processing pipeline may then determine a gradient for each neighboring pixel in the modified set of neighboring pixels and determine whether the input pixel includes a dynamic defect or a speckle based at least in part on the gradient for each neighboring pixel in the modified set of neighboring pixels.

Claims

1. An image signal processing system comprising:

an image sensor configured to receive image data;

an image processing pipeline configured to detect and correct a defective image pixel by;

receiving an input pixel of the image data acquired using the image sensor;

identifying a set of neighboring pixels having the same color component as the input pixel;

removing two neighboring pixels from the set of neighboring pixels thereby generating a modified set of neighboring pixels, wherein the two neighboring pixels correspond to a maximum pixel value and a minimum pixel value of the set of neighboring pixels;

determining a gradient for each neighboring pixel in the modified set of neighboring pixels; and

determining whether the input pixel comprises a speckle based at least in part on the gradient for each neighboring pixel in the modified set of neighboring pixels, wherein determining whether the input pixel comprises the speckle comprises;

determining an average pixel value based at least in part on each pixel value in the modified set of neighboring pixels;

determining an average gradient value based at least in part on a pixel value that corresponds to the input pixel and the average pixel value;

determining a despeckle threshold value by;

determining at least two brightness levels of a plurality of brightness levels, wherein the at least two brightness levels comprise a brightness level above and below a brightness value that corresponds to the average pixel value;

determining a first filter coefficient based at least in part on a first data array, the at least two brightness levels, at least two brightness values associated with the at least two brightness levels, and the average pixel value, wherein the first data array comprises a first plurality of threshold values indexed with respect to a color component of the input pixel and a brightness level of the plurality of brightness levels that corresponds to the average pixel value;

determining a second filter coefficient based at least in part on a second data array, the at least two brightness levels, the at least two brightness values associated with the at least two brightness levels, and the average pixel value, wherein the second data array comprises a second plurality of threshold values indexed with respect to the color component of the input pixel and the brightness level of the plurality of brightness levels that corresponds to the average pixel value; and

interpolating the first filter coefficient and the second filter coefficient;

comparing the average gradient value to the despeckle threshold value; and

identifying the input pixel as comprising the speckle if the average gradient value is greater than the despeckle threshold value.

2. The image signal processing system of claim 1, wherein the image processing pipeline is configured to correct the defective image pixel by:

determining a replacement value for the input pixel if the input pixel comprises the speckle; and

replacing a current value of the input pixel with the replacement value if the input pixel comprises the speckle.

3. The image signal processing system of claim 1, wherein the image processing pipeline is configured to identify the set of neighboring pixels by:

determining the color of the input pixel; and

identifying eight closest neighboring pixels of the same color in an imaging frame, wherein the eight neighboring pixels form a three-by-three pixel block.

4. The image signal processing system of claim 1, wherein the gradient for each neighboring pixel in the modified set of neighboring pixels is a directional pixel-to-pixel gradient calculated with respect to the input pixel and each of the neighboring pixels in the modified set of neighboring pixels.

5. An image signal processing system comprising:

an image processing pipeline configured to detect and correct a defective image pixel by;

receiving an input pixel of image data acquired using an image sensor;

identifying a set of neighboring pixels having the same color component as the input pixel;

removing two neighboring pixels from the set of neighboring pixels thereby generating a modified set of neighboring pixels, wherein the two neighboring pixels correspond to a maximum pixel value and a minimum pixel value of the set of neighboring pixels;

determining a gradient for each neighboring pixel in the modified set of neighboring pixels; and

determining whether the input pixel comprises a dynamic defect based at least in part on the gradient for each neighboring pixel in the modified set of neighboring pixels, wherein determining whether the input pixel comprises the dynamic defect comprises;

comparing each gradient for each neighboring pixel in the modified set of neighboring pixels to an adaptive dynamic defect threshold value, wherein the adaptive dynamic defect threshold value is determined by;

determining a third filter coefficient based at least in part on a third data array, the average pixel value, the at least two brightness levels, and the at least two brightness values associated with the at least two brightness levels, wherein the third data array comprises a third plurality of threshold values indexed with respect to the color component of the input pixel and the brightness level of the plurality of brightness levels that corresponds to the average pixel value;

determining a fourth filter coefficient based at least in part on a fourth data array, the at least two brightness levels, the at least two brightness values associated with the at least two brightness levels, and the average pixel value, wherein the fourth data array comprises a fourth plurality of threshold values indexed with respect to the color component of the input pixel and the brightness level of the plurality of brightness levels that corresponds to the average pixel value; and

interpolating the third filter coefficient and the fourth filter coefficient;

counting a number of gradients that is less than or equal to the adaptive dynamic defect threshold value;

comparing the number to a maximum value; and

determining that the input pixel comprises the dynamic defect if the number is less than or equal to the maximum value.

6. The image signal processing system of claim 5, wherein the maximum value is selected based at least in part on a location of the input pixel.

7. The image signal processing system of claim 5, wherein the adaptive dynamic defect threshold value is determined based at least in part on an average pixel value of the modified set of neighboring pixels, each pixel value in the set of neighboring pixels, and at least two brightness levels in a plurality of brightness levels, wherein the at least two brightness levels comprise a brightness level above and below a brightness value that corresponds to the average pixel value.

8. The image signal processing system of claim 7, wherein the plurality of brightness levels comprise five equally spaced brightness levels defined between 0 and 2^16.

9. The image signal processing system of claim 5, wherein the image processing pipeline is configured to correct the defective image pixel when the input pixel comprises the dynamic defect, wherein the input pixel is corrected based at least in part on a plurality of pixel values that correspond to a 5×

5 array of pixels that surround the input pixel.

10. The image signal processing system of claim 9, wherein the image processing pipeline is configured to correct the defective image pixel by convolving the 5×

5 array of pixels with a symmetric filter that has a 5×

5 spatial support.

11. The image signal processing system of claim 10, wherein the 5×

5 spatial support comprises a plurality of coefficients that correspond to the plurality of pixel values.

12. The image signal processing system of claim 11, wherein the plurality of coefficients comprise eight programmable coefficients.

13. The image signal processing system of claim 12, wherein each programmable coefficient comprises a signed 16-bit number with twelve fractional bits.

14. The image signal processing system of claim 12, wherein each programmable coefficient is determined by comparing the image data and a reference image using a least square fit algorithm, a genetic learning algorithm, a first absolute difference algorithm, or a combination thereof.

15. A method for processing an image, comprising:

receiving an input pixel of image data acquired using an image sensor;

identifying a set of eight neighboring pixels having the same color component as the input pixel;

removing two neighboring pixels from the set of eight neighboring pixels thereby generating a modified set of neighboring pixels, wherein the two neighboring pixels correspond to a maximum pixel value and a minimum pixel value of the set of eight neighboring pixels;

determining an average pixel value based at least in part on each pixel value in the modified set of neighboring pixels;

determining an average gradient value based at least in part on a pixel value that corresponds to the input pixel and the average pixel value;

determining a despeckle threshold value by;

determining at least two brightness levels of a plurality of brightness levels, wherein the at least two brightness levels comprise a brightness level above and below a brightness value that corresponds to the average pixel value;

determining a first filter coefficient based at least in part on a first data array, the at least two brightness levels, at least two brightness values associated with the at least two brightness levels, and the average pixel value, wherein the first data array comprises a first plurality of threshold values indexed with respect to a color component of the input pixel and a brightness level of the plurality of brightness levels that corresponds to the average pixel value;

determining a second filter coefficient based at least in part on a second data array, the at least two brightness levels, the at least two brightness values associated with the at least two brightness levels, and the average pixel value, wherein the second data array comprises a second plurality of threshold values indexed with respect to the color component of the input pixel and the brightness level of the plurality of brightness levels that corresponds to the average pixel value; and

interpolating the first filter coefficient and the second filter coefficient;

comparing the average gradient value to the despeckle threshold value; and

identifying the input pixel as speckled if the average gradient value is greater than the despeckle threshold value.

16. The method of claim 15, wherein interpolating the first filter coefficient and the second filter coefficient comprises:

determining a high frequency component value based at least in part on the average pixel value, each pixel value in the set of eight neighboring pixel values, and a position of the input pixel; and

summing the first filter coefficient and a product of the second filter coefficient and the high frequency component value.

17. The method of claim 15, wherein interpolating the first filter coefficient and the second filter coefficient comprises:

determining a high frequency component value based at least in part on the average pixel value, each pixel value in the set of eight neighboring pixel values, and a position of the input pixel; and

determining a maximum value between the first filter coefficient and a product of the second filter coefficient and the high frequency component value.

18. A system comprising:

a display device for displaying one or more images; and

an image signal processor configured to;

receive an input pixel of image data acquired using an image sensor;

identify a set of neighboring pixels having the same color component as the input pixel;

remove two neighboring pixels from the set of neighboring pixels thereby generating a modified set of neighboring pixels, wherein the two neighboring pixels correspond to a maximum pixel value and a minimum pixel value of the set of neighboring pixels;

determine a gradient for each neighboring pixel in the modified set of neighboring pixels;

determine an adaptive dynamic defect threshold value by;

determining a first filter coefficient based at least in part on a first data array, the at least two brightness levels, at least two brightness values associated with at least two brightness levels in a plurality of brightness levels, wherein the at least two brightness levels comprise a brightness level above and below a brightness value that corresponds to an average pixel value of the modified set of neighboring pixels, and the average pixel value, wherein the first data array comprises a first plurality of threshold values indexed with respect to a color component of the input pixel and a brightness level of the plurality of brightness levels that corresponds to the average pixel value;

determining a second filter coefficient based at least in part on a second data array, the at least two brightness levels, the at least two brightness values associated with the at least two brightness levels, and the average pixel value, wherein the second data array comprises a second plurality of threshold values indexed with respect to the color component of the input pixel and the brightness level of the plurality of brightness levels that corresponds to the average pixel value; and

interpolating the first filter coefficient and the second filter coefficient;

compare each gradient for each neighboring pixel in the modified set of neighboring pixels to the adaptive dynamic defect threshold value;

count a number of gradients that is less than or equal to the adaptive dynamic defect threshold value; and

determine that the input pixel comprises a dynamic defect if the number is less than or equal to a maximum value.

19. The system of claim 18, wherein the image signal processor is configured to interpolate the first filter coefficient and the second coefficient by:

determining a high frequency component value based at least in part on the average pixel value, each pixel value in the set of neighboring pixel values, and a position of the input pixel; and

summing the first filter coefficient and a product of the second filter coefficient and the high frequency component value.

20. The method of claim 18, wherein interpolating the first filter coefficient and the second filter coefficient comprises:

determining a high frequency component value based at least in part on the average pixel value, each pixel value in the set of neighboring pixel values, and a position of the input pixel; and

determining a maximum value between the first filter coefficient and a product of the second filter coefficient and the high frequency component value.

21. An electronic device, comprising:

a display configured to display one or more images;

an image signal processor configured to;

receive an input pixel of image data acquired using an image sensor;

identify a set of neighboring pixels having the same color component as the input pixel;

remove two neighboring pixels from the set of neighboring pixels thereby generating a modified set of neighboring pixels, wherein the two neighboring pixels correspond to a maximum pixel value and a minimum pixel value of the set of neighboring pixels;

determine a gradient for each neighboring pixel in the modified set of neighboring pixels; and

determine whether the input pixel comprises a dynamic defect by;

determining an active dynamic threshold value based at least in part on;

an average pixel value of the modified set of neighboring pixels;

each pixel value in the set of neighboring pixels; and

at least two brightness levels in a plurality of brightness levels, wherein between each adjacent brightness levels of the at least two brightness levels comprise a distinct portion of a set of brightness values that comprises a minimum brightness value and a maximum brightness value associated with the input pixel, and wherein the at least two brightness levels corresponds to a brightness level above and below a brightness value that corresponds to the average pixel value;

comparing each gradient for each neighboring pixel in the modified set of neighboring pixels to the adaptive dynamic defect threshold value;

counting a number of gradients that is less than or equal to the adaptive dynamic defect threshold value;

comparing the number to a maximum value; and

determining that the input pixel comprises the dynamic defect if the number is less than or equal to the maximum value.