DYNAMIC, LOCAL EDGE PRESERVING DEFECT PIXEL CORRECTION FOR IMAGE SENSORS WITH SPATIALLY ARRANGED EXPOSURES

• US 20140211048A1
• Filed: 01/30/2013
• Published: 07/31/2014
• Est. Priority Date: 01/30/2013
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: AltaSens, Inc. (JVC Kenwood Corporation)
• Inventors: Kolli, Naveen
• Granted Patent: US 8,902,336 B2
• Time in Patent Office: Days
• US Class Current: 348/246
• CPC Class Codes: H04N 25/68 applied to defects
• CPC Class Codes: H04N 25/683 by defect estimation performed on the scene signal, e.g. real time or on the fly detection

Abstract

Various technologies described herein pertain to defect pixel correction for image data collected by a pixel array of an image sensor with spatially arranged exposures. The pixel array includes a first subset of pixels having a first exposure time and a second subset of pixels having a second exposure time. An exposure ratio (ratio of first exposure time to second exposure time) is received. A value of at least a particular neighbor pixel of a given pixel from the image data is adjusted based upon the exposure ratio. Neighborhood statistics for the given pixel from the image data are computed based on values of neighbor pixels of the given pixel from the image data as adjusted. Whether the value of the given pixel is defective is detected based on the neighborhood statistics. The value of the given pixel is replaced when detected to be defective to output modified image data.

Claims

1. A defect pixel correction system for image data collected by a pixel array of an image sensor with spatially arranged exposures, comprising:

a neighborhood pre-processing component that receives an exposure ratio for the pixel array and adjusts a value of at least a particular neighbor pixel of a given pixel from the image data based upon the exposure ratio, wherein the exposure ratio is a ratio of a first exposure time of a first subset of pixels of the pixel array to a second exposure time of a second subset of the pixels of the pixel array, wherein neighbor pixels of the given pixel comprise the particular neighbor pixel, wherein the neighbor pixels and the given pixel belong to a particular Bayer domain, and wherein each of the neighbor pixels of the given pixel is one of vertically, horizontally, or diagonally adjacent to the given pixel within the particular Bayer domain;

a neighborhood analysis component that computes neighborhood statistics for the given pixel from the image data based on values of the neighbor pixels of the given pixel from the image data as adjusted by the neighborhood pre-processing component;

a defect detection component that detects whether a value of the given pixel from the image data is defective based upon the neighborhood statistics; and

a replacement component that replaces the value of the given pixel from the image data when detected to be defective to output modified image data.

2. The defect pixel correction system of claim 1, wherein the neighborhood pre-processing component further comprises a membership component that determines whether the given pixel is comprised in one of the first subset of the pixels having the first exposure time or the second subset of the pixels having the second exposure time.

3. The defect pixel correction system of claim 2, wherein the neighborhood pre-processing component further comprises:

an offset component that subtracts a black level from the value of the particular neighbor pixel to compute a remaining value of the particular neighbor pixel; and

a scale component that scales the remaining value of the particular neighbor pixel as a function of the exposure ratio to compute a scaled value of the particular neighbor pixel;

wherein the offset component adds the black level to the scaled value of the particular neighbor pixel to output an adjusted value of the particular neighbor pixel.

4. The defect pixel correction system of claim 3, wherein the scale component scales the remaining value of the particular neighbor pixel as a function of the exposure ratio to compute the scaled value of the particular neighbor pixel by one of:

multiplying the remaining value of the particular neighbor pixel by the exposure ratio when the given pixel is determined to be comprised in the first subset of the pixels having the first exposure time, wherein the first exposure time is greater than the second exposure time;

or

dividing the remaining value of the particular neighbor pixel by the exposure ratio when the given pixel is determined to be comprised in the second subset of the pixels having the second exposure time, wherein the first exposure time is greater than the second exposure time.

5. The defect pixel correction system of claim 1, wherein the neighborhood pre-processing component adjusts the value of the particular neighbor pixel within the particular Bayer domain if and only if:

the first exposure time differs from the second exposure time; and

the particular neighbor pixel is exposed for one of the first exposure time or the second exposure time and the given pixel is exposed for the other one of the first exposure time or the second exposure time.

6. The defect pixel correction system of claim 1, further comprising:

an exception component that detects whether a condition exists when the first exposure time differs from the second exposure time, wherein the exception component, upon detection of the existence of the condition, inhibits replacement of the value of the given pixel from the image data regardless whether detected to be defective;

wherein one of the first exposure time or the second exposure time is a long exposure time and the other one of the first exposure time or the second exposure time is a short exposure time.

7. The defect pixel correction system of claim 6, wherein the condition comprises the neighbor pixels cover a high frequency region, wherein the exception component detects that the neighbor pixels cover the high frequency region when a standard deviation of the values of the neighbor pixels is greater than a product of a high frequency multiplier times a central tendency measure of the values of the neighbor pixels.

8. The defect pixel correction system of claim 6, wherein the condition comprises:

the given pixel has the long exposure time;

the value of the given pixel is above a predefined saturation value; and

the neighbor pixels cover a mid-to-high frequency region, wherein the exception component detects that the neighbor pixels cover the mid-to-high frequency region when a standard deviation of the values of the neighbor pixels is;

greater than a first product of a middle frequency multiplier times a central tendency measure of the values of the neighbor pixels; and

less than or equal to a second product of a high frequency multiplier times the central tendency measure of the values of the neighbor pixels.

9. The defect pixel correction system of claim 6, wherein the condition comprises:

the given pixel has the short exposure time; and

the value of the given pixel is less than a predefined dark threshold.

10. The defect pixel correction system of claim 6, wherein the condition comprises:

the given pixel has the short exposure time; and

at least one of the neighbor pixels having the long exposure time has a value above a predefined saturation value.

11. The defect pixel correction system of claim 1, wherein the defect detection component further comprises:

a statistics evaluation component that detects whether the neighbor pixels cover a flat region or a non-flat region based upon the neighborhood statistics;

a threshold determination component that selects a predefined parameter as a function of whether the neighbor pixels cover the flat region or the non-flat region, wherein the threshold determination component computes a high threshold and a low threshold based upon the selected predefined parameter and the neighborhood statistics; and

a comparison component that compares the value of the given pixel to the high threshold and the low threshold, wherein the value of the given pixel from the image data is detected to be defective when greater than the high threshold or less than the low threshold.

12. The defect pixel correction system of claim 11, wherein the neighborhood statistics comprise a standard deviation of the values of the neighbor pixels and a central tendency measure of the values of the neighbor pixels, and wherein the central tendency measure of the values of the neighbor pixels is one of a mean of the values of the neighbor pixels or a median of the values of the neighbor pixels.

13. The defect pixel correction system of claim 12, wherein the statistics evaluation component detects that the neighbor pixels cover the flat region when the standard deviation of the values of the neighbor pixels is less than a predefined percentage of the central tendency measure of the values of the neighbor pixels, and detects that the neighbor pixels cover the non-flat region when the standard deviation of the values is greater than or equal to the predefined percentage of the central tendency measure of the values of the neighbor pixels.

14. The defect pixel correction system of claim 12, wherein the selected predefined parameter comprises a first multiplier and a second multiplier when the neighbor pixels cover the flat region, wherein the predefined parameter comprises a third multiplier and a fourth multiplier when the neighbor pixels cover a non-flat region, wherein the threshold determination component computes the high threshold as the central tendency measure plus a first product of the standard deviation times the first multiplier and the low threshold as the central tendency measure minus a second product of the standard deviation times the second multiplier when the neighbor pixels cover the flat region, and wherein the threshold determination component computes the high threshold as the central tendency measure plus a third product of the standard deviation times the third multiplier and the low threshold as the central tendency measure minus a fourth product of the standard deviation times the fourth multiplier when the neighbor pixels cover the non-flat region.

15. The defect pixel correction system of claim 12, wherein the replacement component further comprises a flat replacement component that replaces the value of the given pixel from the image data with the central tendency measure of the values of the neighbor pixels to output the modified image data when the neighbor pixels are detected to cover the flat region.

16. The defect pixel correction system of claim 1, wherein the defect detection component further comprises a static identification component that checks whether the given pixel is specified as being defective in a static defect pixel map.

17. The defect pixel correction system of claim 1, wherein the replacement component further comprises:

a neighbor comparison component that respectively compares a subset of the values of the neighbor pixels to a high threshold and a low threshold, wherein the subset of the values of the neighbor pixels have yet to be processed, and wherein a remainder of the values of the neighbor pixels have been previously processed;

a neighbor replacement component that replaces, with a predefined temporary value, the values of the neighbor pixels from the subset that are respectively determined to be greater than the high threshold or less than the low threshold;

an edge computation component that computes magnitudes of a plurality of edges across the given pixel subsequent to replacement of the values of the neighbor pixels from the subset that are respectively determined to be greater than the high threshold or less than the low threshold;

a minimum detection component that identifies a minimum edge having a minimum magnitude from the magnitudes of the plurality of the edges; and

a value exchange component that replaces the value of the given pixel from the image data with an average of the values of the neighbor pixels that belong to the minimum edge to output the modified image data.

18. The defect pixel correction system of claim 17, wherein the edge computation component computes a particular magnitude of a particular edge across the given pixel as an absolute value of a difference between the values of the neighbor pixels that belong to the particular edge.

19. The defect pixel correction system of claim 1 comprised in the image sensor.

20. The defect pixel correction system of claim 1 comprised in an image signal processor.

21. A method of correcting defect pixels in image data collected by a pixel array of an image sensor with spatially arranged exposures, comprising:

receiving an exposure ratio for the pixel array, wherein the exposure ratio is a ratio of a first exposure time of a first subset of pixels of the pixel array to a second exposure time of a second subset of the pixels of the pixel array;

adjusting a value of at least a particular neighbor pixel of a given pixel from the image data based upon the exposure ratio, wherein neighbor pixels of the given pixel comprise the particular neighbor pixel;

computing neighborhood statistics for the given pixel from the image data based on values of the neighbor pixels of the given pixel from the image data as adjusted;

detecting whether a value of the given pixel from the image data is defective based upon the neighborhood statistics; and

replacing the value of the given pixel from the image data when detected to be defective to output modified image data.

22. The method of claim 21, further comprising:

determining whether the given pixel is comprised in one of the first subset of the pixels having the first exposure time or the second subset of the pixels having the second exposure time;

subtracting a black level from the value of the particular neighbor pixel to compute a remaining value of the particular neighbor pixel;

scaling the remaining value of the particular neighbor pixel as a function of the exposure ratio to compute a scaled value of the particular neighbor pixel, wherein the scaled value of the particular neighbor pixel is computed by one of multiplying the remaining value of the particular neighbor pixel by the exposure ratio or dividing the remaining value of the particular neighbor pixel by the exposure ratio; and

adding the black level to the scaled vale of the particular neighbor pixel to output an adjusted value of the particular neighbor pixel.

23. The method of claim 21, wherein the neighbor pixels and the given pixel belong to a particular Bayer domain, and wherein each of the neighbor pixels of the given pixel is one of vertically, horizontally, or diagonally adjacent to the given pixel within the particular Bayer domain.

24. The method of claim 23, further comprising:

adjusting the value of the particular neighbor pixel within the particular Bayer domain if and only if;

the first exposure time differs from the second exposure time; and

the particular neighbor pixel is exposed for one of the first exposure time or the second exposure time and the given pixel is exposed for the other one of the first exposure time or the second exposure time.

25. The method of claim 21, further comprising:

when the first exposure time differs from the second exposure time, detecting whether a condition exists; and

upon detection of the existence of the condition, inhibiting replacement of the value of the given pixel from the image data regardless whether detected to be defective.

26. An image sensor, comprising:

a pixel array;

a buffer that retains image data collected by the pixel array; and

a defect pixel correction system, comprising;

a neighborhood pre-processing component that receives an exposure ratio for the pixel array and adjusts a value of at least a particular neighbor pixel of a given pixel from the image data based upon the exposure ratio, wherein the exposure ratio is a ratio of a first exposure time of a first subset of pixels of the pixel array to a second exposure time of a second subset of the pixels of the pixel array, wherein neighbor pixels of the given pixel comprise the particular neighbor pixel, wherein the neighbor pixels and the given pixel belong to a particular Bayer domain, and wherein each of the neighbor pixels of the given pixel is one of vertically, horizontally, or diagonally adjacent to the given pixel within the particular Bayer domain;

a neighborhood analysis component that computes neighborhood statistics for the given pixel from the image data based on values of the neighbor pixels of the given pixel from the image data as adjusted;

a defect detection component that detects whether a value of the given pixel from the image data is defective based upon the neighborhood statistics;

a replacement component that replaces the value of the given pixel from the image data when detected to be defective to output modified image data; and

an exception component that detects whether a condition exists when the first exposure time differs from the second exposure time, wherein the exception component, upon detection of the existence of the condition, inhibits replacement of the value of the given pixel from the image data regardless whether detected to be defective.