Color interpolator and horizontal/vertical edge enhancer using two line buffer and alternating even/odd filters for digital camera

• US 6,628,330 B1
• Filed: 11/03/1999
• Issued: 09/30/2003
• Est. Priority Date: 09/01/1999
• Status: Expired due to Term

• Current Assignee: Faust Communications LLC (Intellectual Ventures LLC)
• Original Assignee: Neomagic Corporation
• Inventors: Lin, Tao
• Primary Examiner(s): MOE, AUNG SOE
• Application Number:
  US09/432,366
• Time in Patent Office: 1,427 Days
• Field of Search: 348/252, 348/253, 348/254, 348/271, 348/273, 348/279, 348/278, 348/625, 382/266, 382/275, 382/272, 382/300, 382/303, 358/532, 358/518, 358/519, 358/520, 358/525
• US Class Current: 348/252
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: G06T 3/403 Edge-driven scaling; Edge-based scaling
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

A digital-camera processor receives a stream of mono-color pixels in a Bayer pattern from a sensor. Two lines of the pattern are stored in a 2-line buffer. Red, Blue, and Green interpolators receive a 3×3 array of pixels from the 2-line buffer. The interpolators generate missing color values by interpolation. For green, horizontal interpolation is performed for odd lines, while vertical interpolation is performed for even lines. Horizontal and vertical interpolation is thus alternated with alternate lines. Edge detection is performed at the same time as interpolation, on the green pixels from the 2-line buffer. An edge-detection filter is multiplied by the green pixels in the 3×3 array from the 2-line buffer. Different edge-detection filters are used for odd and even lines. These filters are modified to detect edges running perpendicular to the direction of the green interpolation filter. Edges in the same direction as the interpolation filter are ignored. Thus blurring caused by the green interpolation does not affect edge detection. The result of the edge-detection filter is compared to two different threshold values, one for green and one for red and blue. When an edge is detected, an edge enhancer is activated. The edge enhancer adds a scaled factor to the interpolated R, G, or B values to sharpen the detected edge. The line buffer stores only 2 full lines of pixels and no full-frame buffer is needed.

Claims

1. An alternating-line color interpolator and edge detector comprising:

an input for receiving a stream of mono-color pixels from an image sensor;

a 2-line buffer, coupled to receive pixels from the input, for storing a full line of pixels and portions of 2 other lines of pixels, the 2-line buffer outputting a 3×

3 region of pixels having 3 rows and 3 columns;

a red interpolator, receiving red pixels in the 3×

3 region from the 2-line buffer, for averaging the red pixels within the 3×

3 region to generate an interpolated red pixel for a center location of the 3×

3 region;

a blue interpolator, receiving blue pixels in the 3×

3 region from the 2-line buffer, for averaging the blue pixels within the 3×

3 region to generate an interpolated blue pixel for a center location of the 3×

3 region;

a green interpolator, receiving green pixels in the 3×

3 region from the 2-line buffer, for averaging the green pixels within the 3×

3 region to generate an interpolated green pixel for a center location of the 3×

3 region;

wherein for odd lines the green interpolator averages only green pixels in a middle column of the 3×

3 region when the center location is on an odd line, the green interpolator ignoring green pixels in a left column to the left of the middle column and in a right column to the right of the middle column for odd lines;

wherein for even lines the green interpolator averages only green pixels in a middle row of the 3×

3 region when the center location is on an even line, the green interpolator ignoring green pixels in an upper row above the middle row and in a lower row below the middle row for even lines; and

an edge detector, receiving green pixels in the 3×

3 region from the 2-line buffer, for multiplying each green pixel by a filter coefficient and for summing results to generate an edge value;

wherein for odd lines, the filter coefficients for green pixels in a middle column have an opposite sign as filter coefficients for green pixels in the right and left column,

whereby edges crossing the middle column are magnified more than vertical edges within the middle column for odd lines;

wherein for even lines, the filter coefficients for green pixels in a middle row have an opposite sign as filter coefficients for green pixels in the upper and lower row,

whereby edges crossing the middle row are magnified more than horizontal edges within the middle row for even lines,

whereby vertical color interpolation and horizontal edge detection is performed for odd lines, but horizontal color interpolation and vertical edge detection is performed for even lines.

2. The alternating-line color interpolator and edge detector of claim 1 wherein diagonal edges that pass through the center location are detected by the edge detector for both odd and even lines.

3. The alternating-line color interpolator and edge detector of claim 1 wherein the mono-color pixels from the image sensor are arranged in a Bayer pattern.

4. The alternating-line color interpolator and edge detector of claim 3 wherein the 2-line buffer has a storage capacity sufficient for two full lines of pixels but insufficient for three full lines of the pixels;

and

wherein the 2-line buffer stores less than one percent of the whole image of pixels.

5. The alternating-line color interpolator and edge detector of claim 4 wherein the center location is in the middle row and in the middle column.

6. The alternating-line color interpolator and edge detector of claim 5 wherein the green color interpolator averages one or two green pixels for each center location, while the red and blue color interpolators each average one, two, or four red or blue pixels for each center location,

whereby no more than two green pixels are averaged but up to four red or blue pixels are averaged.

7. The alternating-line color interpolator and edge detector of claim 5 wherein the absolute value of the filter coefficient for the center location is at least four times larger than a value of any other of the filter coefficients.

8. The alternating-line color interpolator and edge detector of claim 3 further comprising:

an edge enhancer, receiving the edge value from the edge detector, the edge enhancer altering the interpolated green pixel when an edge absolute value being an absolute value of the edge value exceeds a green threshold, p1 whereby detected edges are enhanced.

9. The alternating-line color interpolator and edge detector of claim 8 wherein the edge enhancer further comprises:

a first comparator for comparing the edge absolute value to the green threshold, for activating a green multiplier when the edge absolute value exceeds the green threshold;

a second comparator for comparing the edge absolute value to a red/blue threshold, for activating a red/blue multiplier when the edge absolute value exceeds the red/blue threshold, the red/blue multiplier altering the interpolated red pixel and the interpolated blue pixel when the edge absolute value exceeds the red/blue threshold,

whereby different edge thresholds are compared for green and for red/blue pixels.

10. The alternating-line color interpolator and edge detector of claim 9 wherein the green multiplier multiplies the edge value with a green scale factor and a green summer adds a product from the green multiplier to the interpolated green pixel for adjustment;

wherein the red/blue multiplier multiplies the edge value with a red/blue scale factor and a red summer adds a product from the red/blue multiplier to the interpolated red pixel while a blue summer adds the product from the red/blue multiplier to the interpolated blue pixel for adjustment,

whereby different scaling factors are used for green and for red/blue.

16. The digital-image processor of claim 5 wherein the odd-line filter detects more vertical and diagonal edges than horizontal edges, while the even line filter detects more horizontal and diagonal edges than vertical edges,

whereby the edge detector is directional for alternate lines.

17. The digital-image processor of claim 16 wherein the mono-color pixels captured by the image sensor form a Bayer pattern, the Bayer pattern having alternating red and green mono-color pixels on odd lines, and alternating green and blue mono-color pixels on even lines, wherein the primary color pixels are green pixels.

11. A digital-image processor comprising:

a line buffer, receiving mono-color pixels captured by an image sensor, the line buffer for storing only a fraction of a whole frame of an image, the fraction being less than 1 percent of a number of pixels in the whole frame;

a merged pipeline, receiving an array of mono-color pixels from the line buffer, for generating missing color values for a middle pixel, the middle pixel being a mono-color pixel in a middle of the array from the line buffer;

wherein for odd lines, primary-color pixels in a middle column of the array are averaged to generate a primary-color pixel in the middle of the array while primary-color pixels in other columns are ignored;

wherein for even lines, primary-color pixels in a middle row of the array are averaged to generate a primary-color pixel in the middle of the array while primary-color pixels in other rows are ignored;

an edge detector, receiving primary-color pixels in the array of mono-color pixels from the line buffer, for detecting an edge between the middle pixel and a neighboring pixel; and

an edge enhancer, activated by the edge detector when the edge is detected, for increasing a magnitude of the middle pixel when the edge is detected;

the merged pipeline outputting interpolated color values for the middle pixel,

whereby interpolation and edge detection are performed by the merged pipeline using the line buffer.

12. The digital-image processor of claim 11 wherein a primary color is green, the primary color pixels are green pixels, and the mono-color pixels are red, green, or blue color values representing a color of a pixel.

13. The digital-image processor of claim 12 wherein the merged pipeline further comprises:

a horizontal interpolator, activated for even lines but not activated for odd lines, receiving the array from the line buffer, for averaging primary-color pixels within a same horizontal line to produce a missing color value of the primary color for the middle pixel; and

a vertical interpolator, activated for odd lines but not activated for even lines, receiving the array from the line buffer, for averaging primary-color pixels within a same vertical column to produce a missing color value of the primary color for the middle pixel;

whereby primary color pixels are horizontally interpolated for even lines but vertically interpolated for odd lines.

14. The digital-image processor of claim 13 wherein the array received by the merged pipeline from the line buffer shifts by one pixel to the right along a line for each clock cycle, the array containing at 3 columns of mono-color pixels from 3 lines;

wherein the line buffer stores 2 lines of mono-color pixels.

15. The digital-image processor of claim 14 wherein the edge detector comprises:

an odd-line filter, activated when the middle pixel is on an odd line, for multiplying each primary-color pixel in the array by a filter coefficient and summing products to generate an edge result;

wherein filter coefficients for the middle column have an opposite sign as filter coefficients for other columns of the array; and

an even-line filter, activated when the middle pixel is on an even line, for multiplying each primary-color pixel in the array by a filter coefficient and summing products to generate an edge result;

wherein filter coefficients for the middle row have an opposite sign as filter coefficients for other rows of the array,

whereby filter coefficients have a vertical arrangement for odd lines but a horizontal arrangement for even lines.

18. An alternating interpolator comprising:

line buffer means, coupled to receive pixels in a Bayer pattern, for storing lines of pixels, the line buffer means outputting a 3×

3 region of pixels having 3 rows and 3 columns;

red interpolate means, receiving red pixels in the 3×

3 region from the line buffer means, for averaging the red pixels within the 3×

3 region to generate an interpolated red pixel for a center location of the 3×

3 region;

blue interpolate means, receiving blue pixels in the 3×

3 region from the line buffer means, for averaging the blue pixels within the 3×

3 region to generate an interpolated blue pixel for a center location of the 3×

3 region;

green interpolate means, receiving green pixels in the 3×

3 region from the line buffer means, for averaging the green pixels within the 3×

3 region to generate an interpolated green pixel for a center location of the 3×

3 region;

green-odd means for averaging green pixels in a middle column of the 3×

3 region when the center location is on an odd line, the green-odd means excluding green pixels in a left column to the left of the middle column and in a right column to the right of the middle column for odd lines;

green-even means for averaging green pixels in a middle row of the 3×

3 region when the center location is on an even line, the green-even means excluding green pixels in an upper row above the middle row and in a lower row below the middle row for even lines; and

edge detect means, receiving green pixels in the 3×

3 region from the line buffer means, for multiplying each green pixel by a filter coefficient and for summing results to generate an edge value;

wherein for odd lines, the filter coefficients for green pixels in a middle column have an opposite sign as filter coefficients for green pixels in the right and left column,

whereby edges crossing the middle column are detected but vertical edges within the middle column are less detected for odd lines;

wherein for even lines, the filter coefficients for green pixels in a middle row have an opposite sign as filter coefficients for green pixels in the upper and lower row,

whereby edges crossing the middle row are detected but horizontal edges within the middle row are less detected for even lines,

whereby vertical color interpolation and horizontal edge detection is performed for odd lines, but horizontal color interpolation and vertical edge detection is performed for even lines.

19. The alternating interpolator of claim 18 wherein the green color interpolate means averages one or two green pixels for each center location, while the red and blue color interpolate means each average one, two, or four red or blue pixels for each center location,

whereby no more than two green pixels are averaged but up to four red or blue pixels are averaged.

20. The alternating interpolator of claim 19 further comprising:

edge enhance means, receiving the edge value from the edge detect means, for altering the interpolated green pixel when an absolute value of the edge value exceeds a green threshold,

whereby detected edges are enhanced.