System and method for detecting and correcting defective pixels in an image sensor

• US 8,259,198 B2
• Filed: 10/20/2009
• Issued: 09/04/2012
• Est. Priority Date: 10/20/2009
• Status: Expired due to Fees

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Cote, Guy, Frederiksen, Jeffrey E.
• Primary Examiner(s): NGUYEN, LUONG TRUNG
• Application Number:
  US12/582,400
• Publication Number:
  US 20110090371A1
• Time in Patent Office: 1,050 Days
• Field of Search: None
• US Class Current: 348/246
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 25/62 Detection or reduction of noise due to excess charges produced by the exposure, e.g. smear, blooming, ghost image, crosstalk or leakage between pixels
• CPC Class Codes: H04N 25/63 applied to dark current
• CPC Class Codes: H04N 25/683 by defect estimation performed on the scene signal, e.g. real time or on the fly detection

Abstract

Various techniques are provided for the detection and correction of defective pixels in an image sensor. In accordance with one embodiment, a static defect table storing the locations of known static defects is provided, and the location of a current pixel is compared to the static defect table. If the location of the current pixel is found in the static defect table, the current pixel is identified as a static defect and is corrected using the value of the previous pixel of the same color. If the current pixel is not identified as a static defect, a dynamic defect detection process includes comparing pixel-to-pixel gradients between the current pixel a set of neighboring pixels against a dynamic defect threshold. If a dynamic defect is detected, a replacement value for correcting the dynamic defect may be determined by interpolating the value of two neighboring pixels on opposite sides of the current pixel in a direction exhibiting the smallest gradient.

Claims

1. A method for detecting and correcting defective image pixels during image processing, comprising:

using an image processor;

receiving an input pixel of image data acquired using an image sensor;

comparing the location of the input pixel within a processing region of an imaging frame of the image sensor to a set of location entries stored in a static defect table;

if the location of the input pixel corresponds to one of the set of location entries, marking the input pixel as a static defect and determining a first replacement value for the input pixel;

if the location of the input pixel does not correspond to one of the set of location entries, identifying a set of neighboring pixels having the same color component as the input pixel, determining whether the input pixel is either a dynamic defect or speckled with respect to the set of neighboring pixels, and determining a second replacement value for the input pixel if the input pixel is determined to be either a dynamic defect or speckled; and

correcting the input pixel, wherein correcting the input pixel comprises;

replacing a current value of the input pixel with the first replacement value if the input pixel is marked as a static defect;

replacing the current value of the input pixel with the second replacement value if the input pixel is determined to be either a dynamic defect or speckled; and

storing the static defect table and the first replacement value in a memory, wherein the static defect table is configured to be updated intermittently to store locations corresponding to newly-detected static defects.

2. The method of claim 1, comprising not correcting the input pixel if the input pixel is not marked as a static defect and is not determined as being a dynamic defect or speckled.

3. The method of claim 1, wherein the first replacement value is determined as the value of a previous pixel of the same color in scan order.

4. The method of claim 1, wherein identifying a set of neighboring pixels comprises determining the color of the input pixel and identifying eight closest neighboring pixels of the same color in the imaging frame, wherein the eight closest neighboring pixels form a three-by-three pixel block;

and

wherein the determining of whether the input pixel is either the dynamic defect or speckled with respect to the set of neighboring pixels is based upon a set of directional pixel-to-pixel gradients calculated with respect to the input pixel for each of the eight closest neighboring pixels that are located within the processing region.

5. The method of claim 4, wherein determining if the input pixel is the dynamic defect comprises:

comparing each of the set of directional pixel-to-pixel gradients to an adaptive dynamic defect threshold value;

based upon the comparison, counting the number of the set of directional pixel-to-pixel gradients that is less than or equal to the adaptive dynamic defect threshold value;

comparing the count to a maximum value, wherein the maximum value is selected based at least partially upon the number of neighboring pixels located within the processing region; and

identifying the input pixel as the dynamic defect if the count is less than or equal to the maximum value.

6. The method of claim 5 wherein the adaptive dynamic defect threshold value comprises:

a fixed component for each color component of the image sensor;

a dynamic component determined based at least partially upon one of an exposure setting or an integration time setting of the image sensor at the time the input pixel was captured; and

a high frequency component calculated based upon the sum of the absolute difference between an average of the values of the eight closest neighboring pixels within the processing region and the value of each neighboring pixel within the processing region;

wherein the adaptive dynamic threshold value is calculated by summing the fixed component corresponding to the color of the input pixel with the product of the dynamic component and the high frequency component.

7. The method of claim 4, wherein determining if the input pixel is speckled comprises:

determining an average gradient value by calculating the difference between the value of the input pixel and an average of the values of the eight closest neighboring pixels within the processing region;

comparing the average gradient value to an adaptive speckle threshold value; and

identifying the input pixel as being speckled if the average gradient value is greater than the adaptive speckle threshold value.

8. The method of claim 7, wherein the adaptive speckle threshold value comprises:

a fixed component for each color component of the image sensor;

a dynamic component determined based at least partially upon a degree of texture in the input pixel and the eight closest neighboring pixels; and

a high frequency component calculated based upon the sum of the absolute difference between an average of the values of the eight closest neighboring pixels within the processing region and the value of each neighboring pixel within the processing region;

wherein the adaptive speckle threshold value is calculated by summing the fixed component corresponding to the color of the input pixel with the product of the dynamic component and the high frequency component.

9. The method of claim 4, wherein determining the second replacement value for the input pixel comprises:

determining a first directional gradient in a horizontal direction as the sum of the absolute difference between the input pixel and each of first and second horizontal neighboring pixels from the set of neighboring pixels;

determining a second directional gradient in a vertical direction as the sum of the absolute difference between the input pixel and each of first and second vertical neighboring pixels from the set of neighboring pixels;

determining a third directional gradient in a diagonal-positive direction as the sum of the absolute difference between the input pixel and each of first and second neighboring pixels in the diagonal-positive direction from the set of neighboring pixels, wherein the diagonal-positive direction corresponds to an upper-right direction;

determining a fourth directional gradient in a diagonal-negative direction as the sum of the absolute difference between the input pixel and each of first and second neighboring pixels in the diagonal-negative direction from the set of neighboring pixels, wherein the diagonal-negative direction corresponds to an upper-left direction;

determining the minimum of the first, second, third, and fourth directional gradients; and

determining the second replacement value by linear interpolation of the values of the first and second neighboring pixels in the direction of the minimum gradient.

10. The method of claim 9, wherein if one of the first or the second neighboring pixel in the direction of the minimum gradient is located outside of a processing frame, the value of the pixel located outside of the processing frame is substituted with the value of the other pixel located within the processing frame during the linear interpolation.

11. An image signal processing circuit comprising:

an interface configured to receive image data acquired using an image sensor of an imaging device; and

an image processing pipeline configured to perform a set of processing operations on the image data, wherein one of the set of processing operations detects and corrects for defective pixels in the image data, and wherein the image processing pipeline comprises defective pixel detection and correction logic configured to;

receive an input pixel of the image data;

determine if the location of the input pixel corresponds to a location entry in a static defect table, identify the input pixel as being a static defect if the location of the input pixel is found in the static defect table, and determine a first replacement value for correcting the input pixel;

if the input pixel is not identified as a static defect, determine if the input pixel is a dynamic defect, and determine a second replacement value for the input pixel if the input pixel is determined to be a dynamic defect, wherein the input pixel is determined to be a dynamic defect by;

identifying eight neighboring pixels of the same color as the input pixel;

calculating pixel-to-pixel gradients with respect to the input pixel for each of the eight neighboring pixels located within a processing region defined within an imaging frame of the image sensor;

determining a count of the number of the pixel-to-pixel gradients which are less than a dynamic defect threshold value, wherein the dynamic defect threshold value is adjustable based upon a high frequency component of the eight neighboring pixels located within the processing region; and

identifying the input pixel as a dynamic defect if the determined count is less than a maximum count value; and

correct the input pixel using either the first or second replacement value based upon whether the input pixel is determined to be a static defect, or a dynamic defect.

12. The image signal processing circuit of claim 11,

wherein the defective pixel detection and correction logic is configured to determine if the input pixel is speckled by calculating an average gradient value as the difference between the value of the input pixel and an average of the values of the eight neighboring pixels within the processing region, and identifying the input pixel as being speckled if the average gradient value is greater than a speckle threshold value.

13. The image signal processing circuit of claim 12, wherein the speckle threshold value is adjustable based upon a high frequency component of the eight neighboring pixels located within the processing region.

14. The image signal processing circuit of claim 13, wherein the speckle threshold value is set more aggressively relative to the dynamic defect threshold value in order to reduce false detection of speckling in areas of the image data having increased texture characteristics.

15. The image signal processing circuit of claim 12, wherein the second replacement value is determined by calculating directional gradients as the sum of the difference between the input pixel and two neighboring pixels of the eight neighboring pixels in each of a horizontal direction, a vertical direction, an upper-right direction, and an upper-left direction, determining the direction corresponding to the minimum gradient, and interpolating the values of the two neighboring pixels associated with the minimum gradient direction.

16. The image signal processing circuit of claim 11, wherein the first replacement value is determined based upon the value of a previous pixel of the same color that immediately precedes the input pixel in scanning order.

17. The image signal processing circuit of claim 11, comprising front-end processing logic configured to perform an initial set of operations on the image data, wherein the front-end processing logic is configured to receive the input pixel of the image data, identify four horizontal neighboring pixels of the same color as the input pixel including two horizontal neighboring pixels located on a first side of the input pixel and two horizontal neighboring pixels located on a second side of the input pixel, determine pixel-to-pixel gradient values with respect to the input pixel for each of the four horizontal neighboring pixels located within a processing region defined within an imaging frame of the image sensor, determine a count of the number of gradient values that is less than or equal to a front-end defective pixel threshold, identify the input pixel as a front-end defect, correct the front-end defect, and collect imaging statistics on the image data after the front-end defect is corrected;

wherein the initial set of operations is performed by the front-end processing logic before the set of operations performed by the image processing pipeline.

18. The image signal processing circuit of claim 17, wherein correcting the front-end defect comprises:

determining the value of a first neighboring pixel having the same color as the input pixel and located to the immediate right of the input pixel in the imaging frame;

determining whether the first neighboring pixel is within the processing region;

replacing the value of the input pixel with the value of the first neighboring pixel if the first neighboring pixel is within the processing region;

if the first neighboring pixel is not within the processing region, determining the value of a second neighboring pixel having the same color as the input pixel and located to the immediate left of the input pixel in the imaging frame; and

replacing the value of the input pixel with the value of the second neighboring pixel.

19. A system, comprising:

an imaging device comprising an image sensor; and

image processing circuitry configured to process image data acquired using the image sensor, wherein the image processing circuitry comprises;

defective pixel detection and correction logic configured to;

determine, for a current image frame of the image data, if the location of an input pixel corresponds to a location entry in a static defect table, identify the input pixel as being a static defect if the location of the input pixel is found in the static defect table, and correct the input pixel using a static defect replacement value; and

if the input pixel is not identified as a static defect, determine if the input pixel is either a dynamic defect or speckled, and correct the input pixel using a non-static defect replacement value if the input pixel is determined to be a dynamic defect or speckled; and

a memory device configured to store the static defect table and the static defect replacement value, wherein the static defect table is configured to be updated intermittently during image processing to store locations corresponding to newly-detected static defects.

20. The system of claim 19, wherein the location of the input pixel of the image data is stored into the static defect table as a newly-detected static defect if, over a particular number of consecutive image frames, the input pixel is repeatedly identified as a dynamic defect.

21. The system of claim 19, wherein the static defect table is implemented as one or more first-in-first-out (FIFO) queues.

22. The system of claim 19, wherein the static defect table is configured to store a plurality of static defect pixels, and wherein the static defect table is configured to be sorted based at least partially upon minimum gradient values associated with the stored plurality of static defect pixels.

23. The system of claim 19, comprising an electronic device, and wherein the imaging device comprises a camera integrated with the electronic device, an external camera configured to couple to an input/output port of the electronic device, or some combination thereof.

24. The system of claim 19, wherein the image sensor comprises a Bayer color filter array, a RGBW color filter array, or a CYGM color filter array.

25. The system of claim 19, wherein a static defect, a dynamic defect, or speckle is the result of the input pixel corresponding to a dead pixel, a stuck pixel, or a leaky pixel on the image sensor.

26. One or more non-transitory tangible computer-readable storage media having instructions encoded thereon for execution by a processor, the instructions comprising:

code to determine if an input pixel in a current frame of image data acquired by an image sensor corresponds to a location entry in a static defect table, and to identify the input pixel as a static defect if the location of the input pixel is found in the static defect table, wherein each location entry in the static defect table is associated with a minimum gradient value;

code to correct the input pixel using a static defect replacement value determined as the value of a previous pixel of the same color in scanning order;

code to, if the input pixel is not identified as a static defect, determine whether the input pixel is a dynamic defect by identifying eight neighboring pixels of the same color as the input pixel, calculating pixel-to-pixel gradients with respect to the input pixel for each of the eight neighboring pixels located within a processing region defined within an imaging frame of the image sensor, determining a count of the number of the gradients which are less than a dynamic defect threshold value, and identifying the input pixel as a dynamic defect if the determined count is less than a maximum count value;

code to calculate a dynamic defect replacement value by calculating directional gradients as the sum of the difference between the input pixel and two neighboring pixels of the eight neighboring pixels in each of a horizontal direction, a vertical direction, an upper-right direction, and an upper-left direction, determining the direction corresponding to the minimum gradient, and interpolating the values of the two neighboring pixels associated with the minimum gradient direction;

code to correct the input pixel using the dynamic defect replacement value;

code to determine if the input pixel was identified as a dynamic defect for a particular number of consecutive frames preceding the current frame and, if the input pixel was identified as a dynamic defect for the particular number of consecutive frames, determine a minimum value of the pixel-to-pixel gradients for each of the eight neighboring pixels located within the processing region, and store the location of the input pixel and the minimum gradient value into the static defect table.

27. The one or more non-transitory tangible computer-readable storage media of claim 26, comprising code to prioritize the order in which static defects are corrected based at least partially upon minimum gradient values associated with the static defects.

28. The one or more non-transitory tangible computer-readable storage media of claim 26, wherein a first time at which the code to determine if the input pixel corresponds to the location entry in the static defect table is executed and a second time at which the code to determine whether the input pixel is a dynamic defect is executed are time-shifted by an amount of time corresponding to the processing of at least one scan line of the image data for the current frame.

29. An image signal processing circuit comprising:

an interface configured to receive image data acquired using an image sensor of an imaging device; and

an image processing pipeline configured to perform a set of processing operations on the image data, wherein one of the set of processing operations detects and corrects for defective pixels in the image data, and wherein the image processing pipeline comprises defective pixel detection and correction logic configured to;

receive an input pixel of the image data;

determine if the location of the input pixel corresponds to a location entry in a static defect table, identify the input pixel as being a static defect if the location of the input pixel is found in the static defect table, and determine a first replacement value for correcting the input pixel;

if the input pixel is not identified as a static defect, determine if the input pixel is a speckled and determine a second replacement value for the input pixel if the input pixel is determined to be speckled, wherein the input pixel is determined to be speckled by;

identifying eight neighboring pixels of the same color as the input pixel;

calculating pixel-to-pixel gradients with respect to the input pixel for each of the eight neighboring pixels located within a processing region defined within an imaging frame of the image sensor;

calculating an average gradient value as the difference between the value of the input pixel and an average of the values of the eight neighboring pixels within the processing region; and

identifying the input pixel as being speckled if the average gradient value is greater than a speckle threshold value, wherein the speckle threshold value is adjustable based upon a high frequency component of the eight neighboring pixels located within the processing region; and

correct the input pixel using either the first or second replacement value based upon whether the input pixel is determined to be a static defect or speckled.