Fast edge directed demosaicing

• US 20050146629A1
• Filed: 01/05/2004
• Published: 07/07/2005
• Est. Priority Date: 01/05/2004
• Status: Active Grant

• Current Assignee: Lifesize, Inc. (Enghouse Systems Limited)
• Original Assignee: Lifesize, Inc. (Enghouse Systems Limited)
• Inventors: Muresan, Darian
• Granted Patent: US 7,525,584 B2
• Time in Patent Office: Days
• US Class Current: 348/280
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

An edge directed demosaicing algorithm for determining an edge direction from an input color filter array (CFA) sampled image is disclosed. Aspects of the present invention include calculating for a current missing green pixel, interpolation errors in an East-West (EW) direction at known neighboring green pixels, and averaging the EW interpolation errors to obtain an EW error. Interpolation errors are also calculated for the current missing green pixel in a North-South (NS) direction at known neighboring green pixels, and the NS interpolation errors are averaged to obtain a NS error. An EW or NS direction indicated by a minimum of the EW error and the NS error is then selected as the edge direction.

Claims

1. A computer-implemented method for determining from an input color filter array (CFA) sampled image, an edge direction, the method comprising:

calculating for a current missing green pixel, interpolation errors in an East-West (EW) direction at known neighboring green pixels, and averaging the EW interpolation errors to obtain an EW error;

calculating for the current missing green pixel, interpolation errors in a North-South (NS) direction at known neighboring green pixels, and averaging the NS interpolation errors to obtain a NS error; and

selecting a direction indicated by a minimum of the EW error and the NS error as the edge direction.

2. The method of claim 1 wherein the selected edge direction, the sampled image, which includes a green channel (G) of green pixels, a red channel (R) of red pixels, and a blue channel (B) of blue pixels, are used to interpolate missing green pixels at red and blue locations in the green channel by:

for the missing green pixel, interpolating a difference image comprising the G-B if the missing green pixel is in a blue location, or G-R if the missing green pixel is in a red location, in the selected edge direction;

in the blue channel, estimating missing blue pixels in green pixel locations using linear interpolation of the blue pixels in the blue channel in the selected edge direction; and

in the red channel, estimating the missing red pixels in blue pixel locations using linear interpolation of the red pixels in the red channel in the selected direction, thereby providing an interpolated full green channel in which all missing green pixels have an interpolated value.

3. The method of claim 2 wherein the sampled image and the full green channel are used to correct the green channel by:

calculating the EW and NS interpolation errors for each pixel in the full green channel;

averaging neighboring EW and NS errors at each green pixel;

selecting the green local edge direction EW or NS based on the EW and NS average errors having a smallest value;

estimating the green pixels at blue pixel locations using linear interpolation of G-B in the selected direction, and wherein the missing blue pixels are interpolated in the selected direction; and

estimating the green pixels at red pixel locations using linear interpolation of G-R in the selected direction, and wherein the missing red pixels are interpolated in the selected direction.

4. The method of claim 2 wherein the sampled image and the full green channel are used to interpolate the red and blue pixels to obtain a quincunx sampling by:

using the full green channel to estimate a local EW or NS edge direction by calculating errors in interpolation in EW and NS directions at each pixel;

labeling each pixel'"'"'s direction as one of EW or NS based on a minimum of the EW and NS average errors in a neighborhood of the current green pixel;

at missing red quincunx samples, interpolating R-G in the direction of the label; and

at missing blue quincunx samples, interpolating B-G in the direction of the label.

5. The method of claim 2 wherein the quincunx sampled red and blue channels and the full green channel are used to interpolate the red and blue pixels to obtain full red and blue channels by:

using the full green channel to estimate a NE or NW edge direction by calculating errors in interpolation in NE and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of NE or NW based on a minimum of NE and NW average errors in a neighborhood of the current green pixel;

at the missing quincunx samples interpolating R-G in the direction of the local label; and

at the missing quincunx samples interpolating B-G in the direction of the local label.

6. The method of claim 2 wherein corrected high density red, green, and blue channels are obtained from the input the full red, green, and blue channels by:

using the G-R channel to estimate a local EW, NE, NS, or NW edge direction by calculating errors in interpolation of the G-R channel in the EW, NE, NS, and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of EW, NE, NS, or NW based on a minimum of the average errors in a neighborhood of the corresponding pixel;

interpolating G-R in the direction of the label;

adding the red channel to the interpolated G-R to obtain a G_R channel;

using the G-B channel to estimate local EW, NE, NS, or NW edge direction by calculating errors in interpolation of the G-B channel in the EW, NE, NS, and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of EW, NE, NS, or NW based on a minimum of the average errors in a neighborhood of the corresponding pixel;

interpolating G-B in the direction of the label;

adding the blue channel to the interpolated G-B to obtain the G_B channel;

averaging the G_R and G_B channels in order to obtain a new green channel (G_new);

calculating R-G_new channel;

using R-G_new to estimate a local EW, NE, NS, or NW edge direction by calculating errors in interpolation in the EW, NE, NS, and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of EW, NE, NS, or NW based on a minimum of the average errors in a neighborhood of the corresponding pixel;

interpolating R-G_new in the direction of the label;

adding the new green channel to the interpolated R-G_new to obtain a new red channel;

calculating B-G_new;

using B-G_new to estimate local EW, NE, NS, or NW edge direction by calculating the errors in interpolation in the EW, NE, NS, and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of EW, NE, NS, or NW based on a minimum of the average errors in a neighborhood of the corresponding pixel;

interpolating B-G_new in the direction of the label; and

adding the new green channel to the interpolated B-G_new to obtain a new blue channel.

7. A computer-implemented method for obtaining corrected high-density red (R), green (G), and blue (B) channels from interpolated red, green, and blue channels from an input color filter array (CFA) sampled image, the method comprising:

calculating a G-R channel;

using the G-R channel to estimate a local EW, NE, NS, or NW edge direction by calculating errors in interpolation of the G-R channel in the EW, NE, NS, and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of EW, NE, NS, or NW based on a minimum of the average errors in a neighborhood of the corresponding pixel;

interpolating G-R in the direction of the label;

adding the red channel to the interpolated G-R to obtain a G_R channel;

calculating the G-B channel;

using the G-B channel to estimate local EW, NE, NS, or NW edge direction by calculating errors in interpolation of the G-B channel in the EW, NE, NS, and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of EW, NE, NS, or NW based on a minimum of the average errors in a neighborhood of the corresponding pixel;

interpolating G-B in the direction of the label;

adding the blue channel to the interpolated G-B to obtain the G_B channel;

averaging the G_R and G_B channels in order to obtain a new green channel (G_new);

calculating R-G_new channel;

using R-G_new to estimate a local EW, NE, NS, or NW edge direction by calculating errors in interpolation in the EW, NE, NS, and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of EW, NE, NS, or NW based on a minimum of the average errors in a neighborhood of the corresponding pixel;

interpolating R-G_new in the direction of the label;

adding the new green channel to the interpolated R-G_new to obtain a new red channel;

calculating B-G_new;

using B-G_new to estimate local EW, NE, NS, or NW edge direction by calculating the errors in interpolation in the EW, NE, NS, and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of EW, NE, NS, or NW based on a minimum of the average errors in a neighborhood of the corresponding pixel;

interpolating B-G_new in the direction of the label; and

adding the new green channel to the interpolated B-G_new to obtain a new blue channel.

8. A computer-readable medium containing program instructions for determining from an input color filter array (CFA) sampled image, an edge direction, the program instructions for:

calculating for a current missing green pixel, interpolation errors in an East-West (EW) direction at known neighboring green pixels, and averaging the EW interpolation errors to obtain an EW error;

calculating for the current missing green pixel, interpolation errors in a North-South (NS) direction at known neighboring green pixels, and averaging the NS interpolation errors to obtain a NS error; and

selecting a direction indicated by a minimum of the EW error and the NS error as the edge direction.

9. The computer-readable medium of claim 8 wherein the selected edge direction, the sampled image, which includes a green channel (G) of green pixels, a red channel (R) of red pixels, and a blue channel (B) of blue pixels, are used to interpolate missing green pixels at red and blue locations in the green channel by:

for the missing green pixel, interpolating a difference image comprising the G-B if the missing green pixel is in a blue location, or G-R if the missing green pixel is in a red location, in the selected edge direction;

in the blue channel, estimating missing blue pixels in green pixel locations using linear interpolation of the blue pixels in the blue channel in the selected edge direction; and

in the red channel, estimating the missing red pixels in blue pixel locations using linear interpolation of the red pixels in the red channel in the selected direction, thereby providing an interpolated full green channel in which all missing green pixels have an interpolated value.

10. The computer-readable medium of claim 9 wherein the sampled image and the full green channel are used to correct the green channel by:

calculating the EW and NS interpolation errors for each pixel in the full green channel;

averaging neighboring EW and NS errors at each green pixel;

selecting the green local edge direction EW or NS based on the EW and NS average errors having a smallest value;

estimating the green pixels at blue pixel locations using linear interpolation of G-B in the selected direction, and wherein the missing blue pixels are interpolated in the selected direction; and

estimating the green pixels at red pixel locations using linear interpolation of G-R in the selected direction, and wherein the missing red pixels are interpolated in the selected direction.

11. The computer-readable medium of claim 9 wherein the input sampled image and the full green channel are used to interpolate the red and blue pixels to obtain a quincunx sampling by:

using the full green channel to estimate a local EW or NS edge direction by calculating errors in interpolation in EW and NS directions at each pixel;

labeling each pixel'"'"'s direction as one of EW or NS based on a minimum of the EW and NS average errors in a neighborhood of the current green pixel;

at missing red quincunx samples, interpolating R-G in the direction of the label; and

at missing blue quincunx samples, interpolating B-G in the direction of the label.

12. The computer-readable medium of claim 9 wherein the quincunx sampled red and blue channels and the full green channel are used to interpolate the red and blue pixels to obtain full red and blue channels by:

using the full green channel to estimate a NE or NW edge direction by calculating errors in interpolation in NE and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of NE or NW based on a minimum of NE and NW average errors in a neighborhood of the current green pixel;

at the missing quincunx samples interpolating R-G in the direction of the local label; and

at the missing quincunx samples interpolating B-G in the direction of the local label.

13. The computer-readable medium of claim 9 wherein corrected high density red, green, and blue channels are obtained from the input the full red, green, and blue channels by:

using the G-R channel to estimate a local EW, NE, NS, or NW edge direction by calculating errors in interpolation of the G-R channel in the EW, NE, NS, and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of EW, NE, NS, or NW based on a minimum of the average errors in a neighborhood of the corresponding pixel;

interpolating G-R in the direction of the label;

adding the red channel to the interpolated G-R to obtain a G_R channel;

using the G-B channel to estimate local EW, NE, NS, or NW edge direction by calculating errors in interpolation of the G-B channel in the EW, NE, NS, and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of EW, NE, NS, or NW based on a minimum of the average errors in a neighborhood of the corresponding pixel;

interpolating G-B in the direction of the label;

adding the blue channel to the interpolated G-B to obtain the G_B channel;

averaging the G_R and G_B channels in order to obtain a new green channel (G_new);

calculating R-G_new channel;

using R-G_new to estimate a local EW, NE, NS, or NW edge direction by calculating errors in interpolation in the EW, NE, NS, and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of EW, NE, NS, or NW based on a minimum of the average errors in a neighborhood of the corresponding pixel;

interpolating R-G_new in the direction of the label;

adding the new green channel to the interpolated R-G_new to obtain a new red channel;

calculating B-G_new;

using B-G_new to estimate local EW, NE, NS, or NW edge direction by calculating the errors in interpolation in the EW, NE, NS, and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of EW, NE, NS, or NW based on a minimum of the average errors in a neighborhood of the corresponding pixel;

interpolating B-G_new in the direction of the label; and

adding the new green channel to the interpolated B-G_new to obtain a new blue channel.

14. An electronic device, comprising:

an input color filter array (CFA) for filtering light at a time of image capture;

a charge coupled device (CCD) for sensing the filitered light to produce a sampled image;

a processor for processing the sampled image; and

an edge directed demosaicing algorithm executed by the processor for;

calculating for a current missing green pixel, interpolation errors in an East-West (EW) direction at known neighboring green pixels, and averaging the EW interpolation errors to obtain an EW error;

calculating for the current missing green pixel, interpolation errors in a North-South (NS) direction at known neighboring green pixels, and averaging the NS interpolation errors to obtain a NS error; and

selecting a direction indicated by a minimum of the EW error and the NS error as the edge direction.

15. The electronic device of claim 14 wherein the selected edge direction, the sampled image, which includes a green channel (G) of green pixels, a red channel (R) of red pixels, and a blue channel (B) of blue pixels, are used to interpolate missing green pixels at red and blue locations in the green channel by:

for the missing green pixel, interpolating a difference image comprising the G-B if the missing green pixel is in a blue location, or G-R if the missing green pixel is in a red location, in the selected edge direction;

in the blue channel, estimating missing blue pixels in green pixel locations using linear interpolation of the blue pixels in the blue channel in the selected edge direction; and

in the red channel, estimating the missing red pixels in blue pixel locations using linear interpolation of the red pixels in the red channel in the selected direction, thereby providing an interpolated full green channel in which all missing green pixels have an interpolated value.

16. The electronic device of claim 15 wherein the sampled image and the full green channel are used to correct the green channel by:

calculating the EW and NS interpolation errors for each pixel in the full green channel;

averaging neighboring EW and NS errors at each green pixel;

selecting the green local edge direction EW or NS based on the EW and NS average errors having a smallest value;

estimating the green pixels at blue pixel locations using linear interpolation of G-B in the selected direction, and wherein the missing blue pixels are interpolated in the selected direction; and

estimating the green pixels at red pixel locations using linear interpolation of G-R in the selected direction, and wherein the missing red pixels are interpolated in the selected direction.

17. The electronic device of claim 15 wherein the sampled image and the full green channel are used to interpolate the red and blue pixels to obtain a quincunx sampling by:

using the full green channel to estimate a local EW or NS edge direction by calculating errors in interpolation in EW and NS directions at each pixel;

labeling each pixel'"'"'s direction as one of EW or NS based on a minimum of the EW and NS average errors in a neighborhood of the current green pixel;

at missing red quincunx samples, interpolating R-G in the direction of the label; and

at missing blue quincunx samples, interpolating B-G in the direction of the label.

18. The electronic device of claim 15 wherein the quincunx sampled red and blue channels and the full green channel are used to interpolate the red and blue pixels to obtain full red and blue channels by:

using the full green channel to estimate a NE or NW edge direction by calculating errors in interpolation in NE and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of NE or NW based on a minimum of NE and NW average errors in a neighborhood of the current green pixel;

at the missing quincunx samples interpolating R-G in the direction of the local label; and

at the missing quincunx samples interpolating B-G in the direction of the local label.

19. The electronic device of claim 15 wherein corrected high density red, green, and blue channels are obtained from the input the full red, green, and blue channels by:

using the G-R channel to estimate a local EW, NE, NS, or NW edge direction by calculating errors in interpolation of the G-R channel in the EW, NE, NS, and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of EW, NE, NS, or NW based on a minimum of the average errors in a neighborhood of the corresponding pixel;

interpolating G-R in the direction of the label;

adding the red channel to the interpolated G-R to obtain a G_R channel;

using the G-B channel to estimate local EW, NE, NS, or NW edge direction by calculating errors in interpolation of the G-B channel in the EW, NE, NS, and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of EW, NE, NS, or NW based on a minimum of the average errors in a neighborhood of the corresponding pixel;

interpolating G-B in the direction of the label;

adding the blue channel to the interpolated G-B to obtain the G_B channel;

averaging the G_R and G_B channels in order to obtain a new green channel (G_new);

calculating R-G_new channel;

using R-G_new to estimate a local EW, NE, NS, or NW edge direction by calculating errors in interpolation in the EW, NE, NS, and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of EW, NE, NS, or NW based on a minimum of the average errors in a neighborhood of the corresponding pixel;

interpolating R-G_new in the direction of the label;

adding the new green channel to the interpolated R-G_new to obtain a new red channel;

calculating B-G_new;

using B-G_new to estimate local EW, NE, NS, or NW edge direction by calculating the errors in interpolation in the EW, NE, NS, and NW directions at each pixel;

labeling each pixel'"'"'s direction as one of EW, NE, NS, or NW based on a minimum of the average errors in a neighborhood of the corresponding pixel;

interpolating B-G_new in the direction of the label; and

adding the new green channel to the interpolated B-G_new to obtain a new blue channel.