Method and apparatus for real time identification and correction of pixel defects for image sensor arrays

• US 20030222995A1
• Filed: 06/04/2002
• Published: 12/04/2003
• Est. Priority Date: 06/04/2002
• Status: Active Grant

• Current Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Original Assignee: Micron Technology, Inc.
• Inventors: Subbotin, Igor, Kaplinsky, Michael
• Granted Patent: US 7,202,894 B2
• Time in Patent Office: Days
• US Class Current: 348/247
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: H04N 25/683 by defect estimation performed on the scene signal, e.g. real time or on the fly detection

Abstract

An image processing system and method compares each pixel of an image obtained from an image sensor array with at least eight surrounding pixels of the same color in the filter array. If the signal of a given pixel is larger than the respective signals of all eight surrounding pixels of the same color, then the value of that central pixel signal is substituted with the maximum signal value among the surrounding eight pixels of the same color. Similarly, if the signal of a given pixel is smaller than the respective signals of all eight surrounding pixels of the same color, then the value of that central pixel signal is substituted with the minimum signal value among the surrounding eight pixels of the same color.

Claims

1. An apparatus for correcting pixel image data comprising:

a memory structure for receiving a plurality of lines of image data from an image obtained from an image sensor array, each line containing signal value data corresponding to a line of pixels in the array; and

a correction section which receives the image data from the memory structure and in which the signal value of a tested pixel in the array is compared with the signal values of at least eight surrounding pixels, and which, if the signal value of the tested pixel is larger than all of the signal values of the eight surrounding pixels, the signal value of the tested pixel is replaced with the largest signal value from among the eight surrounding pixels, and which, if the signal value of the pixel being tested is smaller than all of the signal values of the eight surrounding pixels, the signal value of the tested pixel is replaced with the smallest signal value from among the eight surrounding pixels.

2. The apparatus according to claim 1, wherein the eight surrounding pixels are immediately adjacent to the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area for a monochrome image.

3. The apparatus according to claim 1, wherein the eight surrounding pixels are spaced one pixel away from the tested pixel in the horizontal, vertical and diagonal directions in a two dimensional area, and wherein the eight surrounding pixels are of the same color in a color filter array as the tested pixel.

4. The apparatus according to claim 1, further comprising a controller for controlling the memory structure to read out the image data to the correction section so as to sequentially test the signal value of each pixel in a line of image data.

5. The apparatus according to claim 4, wherein the controller further controls an input of image data into the memory structure such that a new line of image data is being written into the memory structure to replace an oldest line of image data as the oldest line of image data is being read out, and upon completion of testing the signal value of each pixel in a line of image data, further controls the memory structure to read out the updated image data so that the correction section sequentially tests the signal values for the pixels in a next line of image data.

6. The apparatus of claim 1, further comprising an image processing section which performs an image processing operation on the tested pixel of the image data after the tested pixel has been tested.

11. An apparatus for correcting pixel image data, comprising:

a plurality of memory banks each for receiving and temporarily storing a line of image data of an image obtained from an image sensor array, each line containing signal value data respectively corresponding to a line of pixels in the array;

a plurality of sets of shift registers for each receiving the image data from the plurality of memory banks;

a correction circuit for testing the signal value of a pixel to be tested in the image data in one of the sets of shift registers to determine if the signal value of the tested pixel is defective, and if found to be defective, for correcting the defective signal value.

7. The apparatus of claim 16, wherein the image processing circuit performs a color processing operation.

8. The apparatus of claim 16, wherein the image processing circuit performs a color interpolation operation.

9. The apparatus of claim 16, wherein the image processing circuit performs a white balancing operation.

10. The apparatus of claim 16, wherein the image processing circuit performs a sharpness filtering operation.

12. The apparatus of claim 11, wherein the correction circuit tests the pixel to be tested by comparing a signal value of the pixel to be tested with the respective signal values of at least eight pixels surrounding the tested pixel and having the same color type as the tested pixel.

13. The apparatus according to claim 12, wherein the plurality of memory banks includes three memory banks and wherein the eight surrounding pixels are immediately adjacent to the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area for a monochrome image.

14. The apparatus according to claim 12, wherein the plurality of memory banks includes five memory banks and wherein the eight surrounding pixels are spaced one pixel away from the tested pixel in the horizontal, vertical and diagonal directions in a two dimensional area, and wherein the eight surrounding pixels are of the same color in a color filter array as the tested pixel.

15. The apparatus of claim 12, wherein the correction circuit redefines the signal value of the tested pixel to be equal to the maximum signal value from among the signal values of the eight surrounding pixels if the signal value of the tested pixel is larger than all of the signal values for the eight surrounding pixels.

16. The apparatus of claim 12, wherein the correction circuit redefines the signal value of the tested pixel to be equal to the minimum signal value from among the signal values of the eight surrounding pixels if the signal value of the tested pixel is smaller than all of the signal values for the eight surrounding pixels.

17. The apparatus according to claim 11, further comprising:

a delay register for inputting a line of image data directly from the image sensor array into one of the plurality of sets of shift registers in column synchrony with the image data being inputted into the shift registers from the memory banks.

18. The apparatus in accordance with claim 11, further comprising:

an input multiplexer for managing the writing of image data into each of the memory banks;

an output multiplexer for managing the reading out of the image data from the memory banks into the shift registers;

an address generator for controlling the addresses of the data being written into and read out of the memory banks.

19. The apparatus according to claim 11, further comprising a controller for controlling the output of the image data from the plurality of memory banks so that the testing and correcting, if necessary, are performed sequentially for the signal values for each pixel in a line of image data.

20. The apparatus according to claim 19, wherein the controller further controls an input of image data into the memory banks to write a new line of image data into the memory bank containing an oldest line of image data so as to overwrite the oldest line of image data as it is being outputted from the relevant memory bank, and upon completion of testing the signal value of each pixel in a line of image data, further controls the output of the updated image data from the plurality of memory bank so that the testing and correcting, if necessary, are performed sequentially for the signal values for each pixel in a next line of image data.

21. The apparatus of claim 11, further comprising

an image processing circuit for performing an image processing operation on a pixel of the image data already tested, the image processing circuit using a portion of the shift registers which is offset from the shift registers in which the tested pixel has been tested.

22. The apparatus of claim 21, wherein the image processing circuit performs a color processing operation.

23. The apparatus of claim 21, wherein the image processing circuit performs a color interpolation operation.

24. The apparatus of claim 21, wherein the image processing circuit performs a white balancing operation.

25. The apparatus of claim 21, wherein the image processing circuit performs a sharpness filtering operation.

26. A method for correcting pixel image data, comprising:

a) obtaining at least three lines of image data containing signal values for a corresponding number of lines of an image obtained from a pixel array in an image sensor;

b) comparing a signal value of a pixel to be tested from among the at least three lines of signal values with the respective signal values of at least eight pixels from among the at least three lines of signal values and which surround the tested pixel, having the eight surrounding pixels the same color type as the tested pixel;

c) if the signal value of the tested pixel is larger than all of the signal values for the eight surrounding pixels, redefining the signal value of the tested pixel to be equal to the maximum signal value from among the signal values of the eight surrounding pixels;

d) if the signal value of the tested pixel is smaller than all of the signal values for the eight surrounding pixels, redefining the signal value of the tested pixel to be equal to the minimum signal value from among the signal values of the eight surrounding pixels.

27. The method according to claim 26, wherein the eight surrounding pixels are immediately adjacent to the tested pixel in the horizontal, vertical and diagonal directions in a two dimensional area for a monochrome image.

28. The method according to claim 26, wherein the eight surrounding pixels are spaced one pixel away from the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area, and wherein the eight surrounding pixels are of the same color in a color filter array as the tested pixel.

29. The method according to claim 26, wherein the comparing act is not performed for the signal values of the pixels from the outermost two rows and columns in the image sensor pixel array.

30. The method according to claim 30, further comprising

e) repeating acts a) through d) for a next pixel in the same line of image data as the tested pixel, until all the pixels in the line have been tested.

31. The method according to claim 30, further comprising:

f) obtaining a new line of signal values for an new line of image data from the image sensor pixel array; and

g) repeating act e) on a next line of image data and also repeating act f) until all the signal values for each pixel in each line of image data has been tested.

32. The method according to claim 31, further comprising

verifying that no cluster of defective pixels is present after testing each pixel in each line of image data by

h) selecting a first pixel from among the tested image data;

i) selecting a second pixel from among the closest surrounding pixels to the first pixel and which is of the same color type as the first pixel;

j) comparing the difference in signal values between the first pixel and second pixel;

k) comparing the absolute value of the difference to a threshold value, wherein a cluster of defective pixels is determined to be present if the absolute value of the difference is greater than the threshold value; and

l) repeating act h) through k) for each pixel in the tested image data.

33. A method for correcting pixel image data, comprising:

writing at least three lines of image data containing signal values for a corresponding number of lines of an image obtained from a pixel array in an image sensor, into a plurality of memory banks;

sequentially inputting the image data from the memory banks into a plurality of sets of shift registers;

comparing a signal value of a pixel to be tested in the shift registers with the signal values of at least eight surrounding pixels in the shift registers and which are of a same color filter type as the tested pixel;

if the signal value of the pixel being tested is larger than all of the signal values of the eight surrounding pixels, correcting the signal value of the pixel being tested to be equal to the largest signal value from among the eight surrounding pixels; and

if the signal value of the pixel being tested is smaller than all of the signal values of the eight surrounding pixels, correcting the signal value of the pixel being tested to be equal to the smallest signal value from among the eight surrounding pixels.

34. The method according to claim 33, further comprising inputting a line of image data directly into a set of shift registers separately from the plurality of memory banks, but in column synchrony with the lines of image data being inputted into the shift registers from the memory banks, wherein a portion the signal values of the at least eight surrounding pixels for being compared with the signal value of the tested pixel are selected from among the image data inputted directly into the shift registers.

35. The method according to claim 33, wherein the eight surrounding pixels are immediately adjacent to the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area for a monochrome image.

36. The method according to claim 33, wherein the eight surrounding pixels are spaced one pixel away from the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area, and wherein the eight surrounding pixels are of the same color in a color filter array as the pixel being tested.

37. The method according to claim 33, further comprising, upon correcting the signal value of the tested pixel, replacing the corrected value in the corresponding location in the corresponding memory bank.

38. The method according to claim 33, further comprising, upon correcting the signal value of the tested pixel, replacing the corrected value in the corresponding location in the shift registers.

39. The method according to claim 33, further comprising

controlling an input of lines of image data into the memory banks to write a new line of image data into a memory bank containing an oldest line of data so as to overwrite the oldest line of image data as it is being outputted from the relevant memory bank;

and

upon completion of testing the signal value of each pixel in a line of image data, controlling the output of the updated image data from the plurality of memory banks so that the comparing act and the correcting act, if necessary, are performed sequentially for the signal values for each pixel in a next line of image data.

40. The method according to claim 33, further comprising controlling the output of the image data from the memory banks so that the comparing act and the correction act, if necessary, are performed sequentially for the signal values for each pixel in a line of image data.

41. The method according to claim 40, further comprising

replacing the oldest line of image data in the relevant memory bank with a new line of image data, and

repeating the controlling act on a next line of image data and also repeating the replacing act until all the signal values for each pixel in each line of image data in the image has been tested.

42. The method according to claim 33, further comprising, after the comparing act and, if necessary, the correcting act have been performed for the tested pixel, performing a subsequent image processing function using an offset portion of the shift registers from those used to test the tested pixel.

43. The method according to claim 42, wherein the subsequent image processing function is a color processing function.

44. The method according to claim 42, wherein the subsequent image processing function is a color interpolation function.

45. The method according to claim 42, wherein the subsequent image processing function is a sharpness filtering function.

46. The method according to claim 42, wherein the subsequent image processing function is a white balancing function

47. An apparatus for correcting pixel image data comprising:

a plurality of memory banks each for receiving and temporarily storing a line of image data of an image obtained from an image sensor array, each line containing signal value data corresponding to a line of pixels in the array; and

a correction section which receives the image data from the plurality of memory banks, compares the signal value of a tested pixel in the array with the signal values of at least eight surrounding pixels, and replaces the signal value of the tested pixel with the largest signal value from among the eight surrounding pixels if the value of the tested pixel is larger than all of the signal values of the eight surrounding pixels.

48. The apparatus according to claim 47, comprising three memory banks and wherein the eight surrounding pixels are immediately adjacent to the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area for a monochrome image.

49. The apparatus according to claim 47, comprising five memory banks and wherein the eight surrounding pixels are spaced one pixel away from the tested pixel in the horizontal, vertical and diagonal directions in a two dimensional area, and wherein the eight surrounding pixels are of the same color in a color filter array as the tested pixel.

50. An apparatus for correcting pixel image data comprising:

a plurality of memory banks each for receiving and temporarily storing a line of image data of an image obtained from an image sensor array, each line containing signal value data corresponding to a line of pixels in the array; and

a correction section which receives the image data from the plurality of memory banks, compares the signal value of a tested pixel in the array with the signal values of at least eight surrounding pixels, and replaces the signal value of the tested pixel with the smallest signal value from among the eight surrounding pixels if the value of the tested pixel is smaller than all of the signal values of the eight surrounding pixels.

51. The apparatus according to claim 50, comprising three memory banks and wherein the eight surrounding pixels are immediately adjacent to the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area for a monochrome image.

52. The apparatus according to claim 50, comprising five memory banks and wherein the eight surrounding pixels are spaced one pixel away from the tested pixel in the horizontal, vertical and diagonal directions in a two dimensional area, and wherein the eight surrounding pixels are of the same color in a color filter array as the tested pixel.

53. A method for correcting pixel image data comprising:

a) obtaining at least three lines of image data containing signal values for a corresponding number of lines of image data of an image obtained from a pixel array in an image sensor;

b) comparing a signal value of a pixel to be tested from among the at least three lines of image data with the respective signal values of at least eight pixels from among the at least three lines of image data and which surround the tested pixel, the eight surrounding pixels having the same color type as the tested pixel; and

c) replacing the signal value of the tested pixel with the maximum signal value from among the signal values of the eight surrounding pixels if the signal value of the tested pixel is larger than all of the signal values for the eight surrounding pixels.

54. The method according to claim 53, wherein the eight surrounding pixels are immediately adjacent to the tested pixel in the horizontal, vertical and diagonal directions in a two dimensional area for a monochrome image.

55. The method according to claim 53, wherein the eight surrounding pixels are spaced one pixel away from the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area, and wherein the eight surrounding pixels are of the same color in a color filter array as the tested pixel.

56. A method for correcting image data due to defective pixels in an image sensor array, comprising:

a) obtaining at least three lines of image data containing signal values for a corresponding number of lines of image data of an image obtained from a pixel array in an image sensor;

b) comparing a signal value of a pixel to be tested from among the at least three lines of image data with the respective signal values of at least eight pixels from among the at least three lines of image data and which surround the tested pixel, the eight surrounding pixels having the same color type as the tested pixel; and

c) replacing the signal value of the tested pixel with the minimum signal value from among the signal values of the eight surrounding pixels if the signal value of the tested pixel is smaller than all of the signal values for the eight surrounding pixels.

57. The method according to claim 56, wherein the eight surrounding pixels are immediately adjacent to the tested pixel in the horizontal, vertical and diagonal directions in a two dimensional area for a monochrome image.

58. The method according to claim 56, wherein the eight surrounding pixels are spaced one pixel away from the pixel being tested in the horizontal, vertical and diagonal directions in a two dimensional area, and wherein the eight surrounding pixels are of the same color in a color filter array as the tested pixel.

59. An apparatus for correcting pixel image data, comprising:

a plurality of memory banks each for receiving and temporarily storing a line of image data from an image sensor array, each line containing signal values corresponding to a line of pixels in the array; and

a correction circuit which identifies and corrects signal values produced by defective pixels in the array without using a predetermined and stored map of defective pixel locations and without use of threshold values.

60. An image processing apparatus comprising:

an image sensing unit for obtaining an image and outputting an image signal which includes pixel image data containing signal values for each line of the image;

an image processor for processing the image signal; and

a controller for controlling the image sensing unit and the image processor, and

wherein the image processor includes a defective pixel data correction circuit which receives the pixel image data and compares the signal value of a tested pixel with the signal values of at least eight pixels surrounding the tested pixel in the image, and which, if the signal value of the tested pixel is larger than all of the signal values of the eight surrounding pixels, replaces the signal value of the tested pixel with the largest signal value from among the eight surrounding pixels, and which, if the signal value of the tested pixel is smaller than all of the signal values of the eight surrounding pixels, replaces the signal value of the tested pixel with the smallest signal value from among the eight surrounding pixels.

61. A processing system, comprising:

a processor for receiving an processing image data; and

an image data generator for supplying image data to the processor, the image data generator comprising

an image sensing unit for obtaining an image and outputting an image signal,

an image processor for processing the image signal, and

a controller for controlling the image sensing unit and the image processor,

wherein the image processor includes a defective pixel data correction circuit which receives the pixel image data and compares the signal value of a tested pixel with the signal values of at least eight pixels surrounding the tested pixel in the image, and which, if the signal value of the tested pixel is larger than all of the signal values of the eight surrounding pixels, replaces the signal value of the tested pixel with the largest signal value from among the eight surrounding pixels, and which, if the signal value of the tested pixel is smaller than all of the signal values of the eight surrounding pixels, replaces the signal value of the tested pixel with the smallest signal value from among the eight surrounding pixels.