Merged pipeline for color interpolation and edge enhancement of digital images

• US 6,642,962 B1
• Filed: 09/01/1999
• Issued: 11/04/2003
• Est. Priority Date: 09/01/1999
• Status: Expired

• Current Assignee: Faust Communications LLC (Intellectual Ventures LLC)
• Original Assignee: Neomagic Corporation
• Inventors: Hwang, Beong-Kwon, Yu, Vincent Chor-Fung, Lin, Tao, Tang, Tianhua
• Primary Examiner(s): MOE, AUNG SOE
• Application Number:
  US09/388,318
• Time in Patent Office: 1,525 Days
• Field of Search: 348/252, 348/253, 348/254, 348/271, 348/272, 348/273, 348/279, 348/278, 348/625, 348/223.1, 348/222.1, 358/532, 358/578, 358/519, 358/520, 358/525, 358/516, 382/266, 382/275, 382/272, 382/300, 382/303
• US Class Current: 348/252
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: G06T 3/403 Edge-driven scaling; Edge-based scaling
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

A digital-camera processor receives mono-color digital pixels from an image sensor. Each mono-color pixel is red, blue, or green. The stream of pixels from the sensor has alternating green and red pixels on odd lines, and blue and green pixels on even lines in a Bayer pattern. Each mono-color pixel is white balanced by multiplying with a gain determined in a previous frame and then stored in a line buffer. A horizontal interpolator receives an array of pixels from the line buffer. The horizontal interpolator generates missing color values by interpolation within horizontal lines in the array. The intermediate results from the horizontal interpolator are stored in a column buffer, and represent one column of pixels from the line buffer. A vertical interpolator generates the final RGB value for the pixel in the middle of the column register by vertical interpolation. The RGB values are converted to YUV. The vertical interpolator also generates green values for pixels above and below the middle pixel. These green values are sent to an edge detector. The edge detector applies a filter to the 3 green values and 6 more green values from the last 2 clock cycles. When an edge is detected, an edge enhancer is activated. The edge enhancer adds a scaled factor to the Y component to sharpen the detected edge. Color enhancement is performed on the U and V components. The line buffer stores only 4 full lines of pixels and no full-frame buffer is needed.

Claims

1. A digital-image processor comprising:

a line buffer, receiving mono-color pixels captured by an image sensor, the line buffer for storing only a fraction of a whole frame of an image, the fraction being less than 5 percent of a number of pixels in the whole frame;

a merged pipeline, receiving an array of mono-color pixels from the line buffer, for generating missing color values for a middle pixel, the middle pixel being a mono-color pixel in a middle of the array from the line buffer;

the merged pipeline also for generating an upper primary color value for an upper pixel immediately above the middle pixel and a lower primary color value for a lower pixel immediately below the middle pixel;

an edge detector, receiving the upper primary color value and the lower primary color value and a primary color value for the middle pixel, the edge detector for detecting an edge between the middle pixel and a neighboring pixel; and

an edge enhancer, activated by the edge detector when the edge is detected, for increasing a magnitude of the middle pixel when the edge is detected;

the merged pipeline outputting interpolated color values for the middle pixel,

whereby interpolation and edge detection are performed by the merged pipeline using the line buffer.

2. The digital-image processor of claim 1 further comprising:

a color-space converter, coupled to receive the middle pixel including the missing color values generated by the merged pipeline, for converting color values into luminance and chromatic values;

wherein the edge enhancer increases the luminance value of the middle pixel when activated.

3. The digital-image processor of claim 2 wherein the color-space converter converts from red, green, blue RGB color values to luminance and chromatic YUV values representing a color of a pixel.

4. The digital-image processor of claim 3 wherein the edge detector receives green color values, the edge detector performing edge detection using green values, but the edge enhancer sharpening edges by adjusting the luminance value of the middle pixel.

5. The digital-image processor of claim 1 wherein the merged pipeline further comprises:

a horizontal interpolator, receiving the array from the line buffer, for averaging mono-color pixels within a same horizontal line to produce intermediate color values;

a column register, coupled to receive the intermediate color values from the horizontal interpolator, for storing intermediate color values for a single column of pixels in the array; and

a vertical interpolator, receiving the intermediate color values from the column register, for averaging intermediate color values within a same single column of pixels, the vertical interpolator outputting RGB color values for the middle pixel, the vertical interpolator also outputting G values as the upper and lower primary color values to the edge detector,

wherein horizontal and vertical interpolation are performed separately.

6. The digital-image processor of claim 5 wherein the array received by the merged pipeline from the line buffer shifts by one pixel to the right along a line for each clock cycle, the array containing at 3 columns of mono-color pixels from at least 5 lines;

wherein the line buffer stores 4 lines of mono-color pixels.

7. The digital-image processor of claim 6 wherein the column register includes one green register for each of at least 5 lines, the column register also including at least 3 shared registers for a middle line containing the middle pixel and an upper and a lower line immediately above and below the middle line;

wherein the shared registers alternately store red and blue color values for alternate clock cycles.

8. The digital-image processor of claim 7 wherein the line buffer includes a temporary buffer for storing a portion of a fifth line, the portion being part of the array sent to the merged pipeline.

9. The digital-image processor of claim 5 wherein the mono-color pixels are not stored in a whole-frame buffer, but only in the line buffer and in pipeline registers, wherein all the pipeline registers together store less than a line of pixels,

whereby whole-frame buffers are not needed.

10. The digital-image processor of claim 5 wherein the mono-color pixels captured by the image sensor form a Bayer pattern, the Bayer pattern having alternating red and green mono-color pixels on odd lines, and alternating green and blue mono-color pixels on even lines.

11. The digital-image processor of claim 5 wherein the fraction of the whole frame of the image that is stored by the line buffer is less than one percent of the whole frame,

whereby the line buffer is 99 percent smaller than a whole-frame buffer.

12. The digital-image processor of claim 5 further comprising:

a white-balance multiplier, coupled to receive the mono-color pixels from the image sensor, for multiplying red pixels by a red gain, for multiplying green pixels by a green gain, and for multiplying blue pixels by a blue gain, the white-balance multiplier outputting white-balanced mono-color pixels to the line buffer,

whereby the mono-color pixels are white-balanced on-the-fly before being written to the line buffer.

13. A method for processing digital images comprising:

receiving a series of mono-color pixels, each pixel representing one of three primary colors;

white-balancing the mono-color pixels by multiplying each mono-color pixel by a color-specific gain to produce balanced pixels;

writing the balanced pixels to a line buffer containing only a few lines of an image;

selecting a range within the line buffer, the range surrounding a target pixel;

horizontally interpolating balanced pixels for each line within the range selected from the line buffer, to produce intermediate components;

storing the intermediate components and some of the mono-color pixels within one column of the range selected from the line buffer in a column register;

vertically interpolating the intermediate components and mono-color pixels from the column register to produce a full-color pixel for the target pixel, and also producing a primary-color component for a pixel above the target pixel and a pixel below the target pixel;

detecting an edge at the target pixel by computing a filter value from the primary-color components of the target pixel and of the pixel above and the pixel below;

when the filter value exceeds a threshold, enhancing the edge by adjusting the target pixel; and

outputting the full-color pixel after adjustment for edge enhancement as the target pixel,

whereby the target pixel is interpolated and edge enhanced using outputs from vertical interpolation.

14. The method of claim 13 further comprising:

converting the full-color pixel to a YUV pixel that includes a Y brightness and U and V color components;

wherein enhancing the edge by adjusting the target pixel comprises adjusting the Y brightness but not the U and V color components of the YUV pixel;

wherein outputting the full-color pixel after adjustment comprises outputting the YUV pixel after Y adjustment for edge enhancement.

15. The method of claim 14 wherein horizontally interpolating balanced pixels comprises:

right-shifting a pair of digital values for mono-color pixels to weight the digital values;

adding right-shifted digital values for mono-color pixels to the immediate right and left of the target pixel to produce an intermediate component,

whereby horizontal interpolation is performed by shifting and adding adjacent mono-color pixels on a line.

16. The method of claim 15 further comprising:

color enhancing the full-color pixel by adjusting the U and V color components of the YUV pixel but not adjusting the Y brightness;

wherein outputting the full-color pixel after adjustment comprises-outputting the YUV pixel after Y adjustment for edge enhancement and U and V adjustment for color enhancement,

whereby color enhancement is performed.

17. A single-pass interpolator and edge detector comprising:

line-buffer means, receiving partial pixels with incomplete color information, for storing at least 2 lines of pixels of an image being processed but storing less than one-tenth of the image;

horizontal interpolate means, receiving a range of pixels from the line-buffer means, for processing pixels within the range through a horizontal filter, the horizontal filter operating only on pixels in a same line, the horizontal interpolate means outputting intermediate results;

column means, coupled to the horizontal interpolate means, for holding the intermediate results from a single column, the intermediate results including color information not present in a partial pixel but interpolated from adjacent pixels in a line;

vertical interpolate means, receiving the intermediate results from the column means, for processing the intermediate results to generate full-color information for a target pixel in the range, the full color information including an edge color, and for generating the edge color but not the full-color information for a pair of surrounding pixels adjacent to the target pixel; and

edge detect means, receiving the edge color for the target pixel and for the pair of surrounding pixels from the vertical interpolate means, for detecting an edge within the image,

whereby full-color information is interpolated from partial pixels while edge detection uses interpolation results without an intermediate full-frame buffer between the vertical interpolate means and the edge detect means.

18. The single-pass interpolator and edge detector of claim 17 further comprising:

color-space conversion means, receiving the full-color information for the target pixel, for converting the full-color information to brightness and hue information for output;

edge enhance means, activated by the edge detect means, for adjusting a brightness of the target pixel when an edge is detected,

whereby edges are detected from the edge color, but brightness is adjusted.

19. The single-pass interpolator and edge detector of claim 18 wherein the column means stores intermediate results from a single column of the range of pixels read from the line-buffer means,

whereby a single column of intermediate results are sent from the horizontal interpolate means to the vertical interpolate means.

20. The single-pass interpolator and edge detector of claim 19 wherein the horizontal interpolate means includes add means for adding color values of adjacent pixels;

wherein the vertical interpolate means includes shift means for weighting color values and add means for adding weighted color values of adjacent pixels.