Image restoration method and associated apparatus

• US 6,493,029 B1
• Filed: 07/08/1998
• Issued: 12/10/2002
• Est. Priority Date: 03/15/1996
• Status: Expired due to Term

• Current Assignee: VLSI Vision Ltd. (STMicroelectronics NV)
• Original Assignee: VLSI Vision Ltd. (STMicroelectronics NV)
• Inventors: Smith, Stewart Gresty, Denyer, Peter Brian
• Primary Examiner(s): Vu, Ngoc-Yen, VU, NGOC YEN T
• Application Number:
  US09/101,378
• Time in Patent Office: 1,616 Days
• Field of Search: 358/518, 358/520, 358/524, 358/482, 358/483, 359/885, 359/891, 382/162, 382/167, 382/190, 382/195, 382/199, 382/205, 348/207, 348/222, 348/223, 348/224, 348/266, 348/272, 348/273, 348/277, 348/280, 348/234, 348/235, 348/236, 348/237, 348/238, 348/222.1
• US Class Current: 348/236
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: G06T 3/403 Edge-driven scaling; Edge-based scaling
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

A signal processing method for estimating luminance is provided, for use with matrix patterned image sensors where elected sensor pixels (G), preferably green pixels, from which luminance-representative signals are taken, only occupy alternate horizontal and vertical positions in the matrix pattern. The method is designed for detecting luminance contour edges and/or stripes in an image so as to enable images obtained form the sensor to be enhanced. Each pixel site (Hc) which is at the centre of both a row and a column of five pixels, and which is not the site of an elected (green) pixel, is identified and, for each such identified site (Hc), the signals output from the immediately adjacent four pixels (Gn,Ge,Gs,Gw) in said row and column are processed to establish both signal amplitude and pixel position. A luminance value for the pixel at the identified site (Hc) is evaluated from the amplitude values (G2,G3) of the median pair of said signal amplitudes in accordance with an algorithm where decision taking is based upon the location of the pixels from which the two signals (G1,G2) of highest amplitude emanate. When the two signals (G1, G2) of highest amplitude emanate from pixels in the same row or column, a further signal processing step is undertaken to identify, in said row and column, the signal outputs from the two respective next adjacent pixels (H1,H4) to the pixels having the highest and lowest signal amplitudes (G1,G4), and the signal output from the pixel at the identified site (Hc), to enable an amplitude comparison to be made according to the result of which one of said median pair (G2,G3) is selected. Signal processing methods for use with colour, monochrome and mixed colour/monochrome image sensors are claimed, as well as a single-chip camera, and also a video camera system, in which the claimed signal processing methods are implemented. the two signals (G1,G2) of highest amplitude emanate from pixels in the same row or column, a

Claims

1. A signal processing method for estimating luminance, for use with matrix patterned image sensors where elected sensor pixels (G) from which luminance-representative signals are taken only occupy alternate horizontal and vertical positions in the matrix pattern, the method comprising identifying each pixel (Hc) which is at the centre of both a row and a column of five pixels and which is not the site of one of said elected sensor pixels (G), and for each such identified pixel (Hc) processing the signals output from the immediately adjacent four pixels (Gn, Ge, Gs, Gw) in said row and column to establish both signal amplitude and pixel position and evaluating a luminance value for the identified pixel from the amplitude values (G2, G3) of a median pair of said signal amplitudes in accordance with an algorithm where decision taking is based upon the location of the pixels from which the two signals (G1, G2) of highest amplitude emanate, the algorithm being such that when the two signals (G1, G2) of highest amplitude emanate from pixels in the same row or column a further signal processing step (7) is undertaken to identify, in said row and column, the signal outputs from the two respective next adjacent pixels (H1, H4) to the pixels having the highest and lowest signal amplitudes (G1, G4), and the signal output from the identified pixel (Hc), to enable an amplitude comparison to be made according to a result of which one of said median pair (G2, G3) is selected.

2. A signal processing method according to claim 1, wherein the selected one of said median pair (G2, G3) is the higher amplitude one (G2) where the signal output from the next adjacent pixel (H1) to said pixel having the highest signal amplitude (G1) is closer in amplitude to the signal output from the pixel at the identified pixel (Hc) than the signal output from the next adjacent pixel (H4) to said pixel having the lowest signal amplitude (G4).

3. A signal processing method according to claim 1, wherein the selected one of said median pair (G2, G3) is the lower amplitude one (G3) where the signal output from the next adjacent pixel (H4) to said pixel having the lowest signal amplitude (G4) is closer in amplitude to the signal output from the pixel at the identified pixel (Hc) than the signal output from the next adjacent pixel (H1) to said pixel having the highest amplitude (G1).

19. A camera (20) comprising a matrix array image sensor (21) and signal processing means (25) formed and arranged for carrying out the signal processing method according to claim 1.

20. A camera (20) according to claim 19, wherein said matrix array image sensor (21) comprises an array of red (R), green (G) and blue (B) pixels, and said elected luminance-sensing pixels are green pixels.

21. A camera (20) according to claim 19, wherein said matrix array image sensor (21) comprises a mixed colour and monochrome array in which monochrome luminance-sensing pixels (Y) are arranged in a chequerboard pattern with colour-sensing pixels (R, G, B).

22. A camera according to claim 19, wherein said matrix array image sensor (21) and said signal processing means (25) are provided together in an integrated circuit.

23. A camera system incorporating a video camera according to claim 19.

4. A signal processing method for estimating luminance for use with matrix array colour image sensors in which elected luminance-sensing pixels (G) capable of producing signals representative of luminance are arranged in a chequerboard pattern with colour sensing pixels (R, B) which are capable of producing signals representative of colour, the method comprising the steps of:

ranking the signal values (G1, G2, G3, G4) representative of luminance obtained from a cluster of four luminance-sensing pixels (Gn, Ge, Gs, Gw) surrounding a first colour-sensing pixel (Hc), analysing the spatial distribution of said ranked signal values (G1, G2, G3, G4) so as to detect the presence of any diagonal luminance contour edge, horizontal luminance contour stripe or vertical luminance contour stripe at said first colour-sensing pixel (Hc), and determining an estimated discrete signal value representative of luminance for said first colour-sensing pixel based on one or more middle ranked signal values (G2, G3) in a contour-enhancing fashion dependent upon, firstly, whether any diagonal luminance contour edge is detected and, secondly, whether any horizontal or vertical luminance contour stripe which is detected is interpreted as a dark stripe on a light background or a light stripe on a dark background, wherein when the horizontal or vertical luminance contour stripe is detected the signal values representative of colour obtained from a plurality of said colour-sensing pixels (R, B) in the vicinity of said cluster of four luminance-sensing pixels (Gn, Ge, Gs, Gw) are compared so as to determine whether said detected horizontal or vertical luminance contour stripe is interpreted as a dark stripe on a light background or a light stripe on a dark background.

5. A signal processing method according to claim 4, wherein said plurality of colour-sensing pixels in the vicinity of said cluster of four luminance-sensing pixels (Gn, Ge, Gs, Gw) includes said first colour-sensing pixel (Hc), a second colour-sensing pixel (H1,H4) disposed within the detected horizontal or vertical luminance contour stripe and a third colour-sensing pixel (H4,H1) disposed outwith the detected horizontal or vertical luminance contour stripe.

6. A signal processing method according to claim 4, wherein the estimated discrete signal value representative of luminance is based on the mean of the middle ranking signal values (G2, G3) when a diagonal contour edge is detected at said first colour-sensing pixel (Hc).

7. A signal processing method according to claim 4, wherein said estimated discrete signal value is based on a corresponding one of the two middle ranking signal values (G2, G3) when a stripe detected at said first colour-sensing pixel (Hc) is interpreted as a dark stripe on a light background or as a light stripe on a dark background.

8. A signal processing method according to claim 7, wherein said corresponding one of the two middle ranking signal values (G2, G3) is the lower ranked one (G3) when a dark stripe on a light background is detected and is the higher ranked one (G2) when a light stripe on a dark background is detected.

9. A signal processing method according to claim 4, wherein a diagonal contour edge is detected when the two highest ranking signal values (G1, G2) are obtained from luminance-sensing pixels (G) disposed diagonally adjacent one another.

10. A signal processing method according to claim 4, wherein a horizontal or vertical contour stripe is detected when the two highest ranking signal values (G1, G2) are obtained from luminance-sensing pixels (G) disposed horizontally or vertically opposite one another.

11. A signal processing method according to claim 4, wherein, where a horizontal or vertical stripe is detected, the signal value obtained from said first colour sensing pixel (Hc) is preferably compared with the signal values obtained from second and third colour-sensing pixels (H1, H4) disposed respectively vertically and horizontally opposite said first colour-sensing pixel (Hc) and immediately adjacent to the luminance-sensing pixels having the highest and lowest ranked signal values (G1, G4), and if the signal value of said first colour-sensing pixel (Hc) is closest in value to the signal value of the pixel (H1) immediately adjacent to said highest-ranked signal value (G1) the stripe is interpreted as a light stripe on a dark background, and if the signal value of said first colour-sensing (Hc) is closest in value to the signal value of the pixel (H4) immediately adjacent to said lowest-ranked signal value (G4) the stripe is interpreted as a dark stripe on a light background.

12. A signal processing method according to claim 4, wherein the steps of the method are repeated for every colour-sensing pixel (R,B) in the matrix array sensor which is not also an elected luminance-sensing pixel (G) and which is at a center of both a row and a column of five pixels.

13. A signal processing method according to claim 4, wherein the estimated discrete signal value representative of luminance obtained from each said colour sensing pixel (R, B) in the array is used to provide a luminance signal for a luminance channel (35, 31) of a camera (20) incorporating an image sensor array (23), which luminance signal is further processed to obtain an “

edge-extracted”

luminance signal which is, in turn, superimposed on lowpass filtered colour channel signals (R, G, B) in the camera (20) so as to provide luminance information for every pixel in the array.

14. A signal processing method for estimating luminance, for use with matrix patterned sensors where elected sensor pixels (P) from which luminance-representative signals are taken only occupy alternate horizontal and vertical matrix element positions in the matrix pattern, the method comprising

identifying each matrix element site (C) which is at the centre of both a row and a column of seven matrix elements and which is not the site of a said elected sensor pixel (P), and for each such identified site (C) processing the signals output from the immediately adjacent four pixels (Yn, Ye, Ys, Yw) in said row and column to establish both signal amplitude and pixel position and evaluating a luminance value for the matrix element of the identified site (C) from the amplitude values of the median pair (Y2, Y3) of said signal amplitudes in accordance with an algorithm where decision taking is based upon the location of the pixels from which the two signals (Y1, Y2) of highest amplitude emanate, the algorithm being such that when the two signals (Y1, Y2) of highest amplitude emanate from pixels in the same row or column a further signal processing step is undertaken to identify outputs (X1, X4) in said row and column from the two pixels respectively next adjacent to the pixels having the highest and lowest signal amplitudes (Y1, Y4), and to identify the mean (M) of signal outputs from the eight pixels (Xc) diagonally adjacent to, and arranged around, said four pixels (Yn, Ye, Ys, Yw) in said row and column, to enable an amplitude comparison to be made according to the result of which one of the said median pair (Y2, Y3) is selected.

15. A signal processing method for estimating luminance for use in a matrix array monochrome image sensor in which elected luminance-sensing pixels (P) capable of producing signals representative of luminance are arranged in a chequerboard pattern with non-luminance-representative matrix array elements (N) which do not produce signals, the method comprising the steps of:

ranking the signal values (Y1, Y2, Y3, Y4) representative of luminance obtained from a cluster of four luminance-sensing pixels (Yn, Ye, Ys, Yw) surrounding a first non-luminance-representative matrix element (C), analysing the spatial distribution of said ranked signal values so as to detect the presence of any diagonal luminance contour edge, horizontal luminance contour stripe or vertical luminance contour stripe at said first non-luminance-representative matrix element (C), and determining an estimated discrete signal value representative of luminance for said non-luminance-representative matrix element (C) based on one or more middle ranked signal values (Y2, Y3) in a contour-enhancing fashion dependent upon, firstly, whether any diagonal luminance contour edge is detected and, secondly, whether any horizontal or vertical luminance contour stripe which is detected is interpreted as a dark stripe on a light background or a light stripe on a dark background, wherein when a horizontal or vertical luminance contour stripe is detected the signal values representative of luminance obtained from at least second and third luminance-sensing pixels (XI, X4) disposed respectively within and outwith the detected horizontal or vertical luminance contour stripe are compared with the mean (M) of the signal values obtained from a plurality of further luminance-sensing pixels (Xc) in the vicinity of said cluster of four luminance-sensing pixels (Yn, Ye, Ys, Yw) so as to determine whether said horizontal or vertical luminance contour stripe is interpreted as a dark stripe on a light background or a light stripe on a dark background.

16. A signal processing method according to claim 15, wherein said plurality of further luminance-sensing pixels (Xc) comprises eight pixels, each pixel being disposed diagonally adjacent one of said cluster of four luminance-sensing pixels (Yn, Ye, Ys, Yw) surrounding said first non-luminance representative matrix element (C).

17. A signal processing method according to claim 15, wherein said second and third luminance-sensing pixels (X1, X4) are disposed horizontally and vertically opposite said first non-luminance representative matrix element (C) and adjacent to the two of the cluster of four luminance-sensing pixels (Yn, Ye, Ys, Yw) which have the highest and lowest ranked signal values (Y1, Y4).

18. A signal processing method according to claim 15, wherein the steps of the method are repeated for each non-luminance representative matrix element in the matrix array which is at the centre of both a row and a column containing seven matrix elements.