SYSTEM AND METHOD FOR DEMOSAICING IMAGE DATA USING WEIGHTED GRADIENTS

• US 20110090242A1
• Filed: 10/20/2009
• Published: 04/21/2011
• Est. Priority Date: 10/20/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Côté, Guy, Frederiksen, Jeffrey E.
• Granted Patent: US 8,638,342 B2
• Time in Patent Office: Days
• US Class Current: 345/597
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 25/611 Correction of chromatic aberration
• CPC Class Codes: H04N 25/683 by defect estimation performed on the scene signal, e.g. real time or on the fly detection

Abstract

Various techniques are provided herein for the demosaicing of images acquired and processed by an imaging system. The imaging system includes an image signal processor and image sensors utilizing color filter arrays (CFA) for acquiring red, green, and blue color data using one pixel array. In one embodiment, the CFA may include a Bayer pattern. During image signal processing, demosaicing may be applied to interpolate missing color samples from the raw image pattern. In one embodiment, interpolation for the green color channel may include employing edge-adaptive filters with weighted gradients of horizontal and vertical filtered values. The red and blue color channels may be interpolated using color difference samples with co-located interpolated values of the green color channel. In another embodiment, interpolation of the red and blue color channels may be performed using color ratios (e.g., versus color difference data).

Claims

1. A method for processing image data, comprising:

using an image processor;

receiving a raw image pattern acquired using an image sensor, the raw image pattern comprising a plurality of green, red, and blue pixels arranged in accordance with a color filter array;

interpolating a green value for each non-green pixel in the image pattern to obtain a full set of green values for the image data, wherein interpolating the green value for each non-green pixel comprises;

receiving a current non-green pixel;

determining a horizontal energy component and a vertical energy component for the current non-green pixel;

applying horizontal filtering to determine a horizontal component of the interpolated green value for the current non-green pixel;

applying vertical filtering to determine a vertical component of the interpolated green value for the current non-green pixel;

applying a first weighting factor to the horizontal component and a second weighting factor to the vertical component, wherein the first weighting factor is based at least partially upon the ratio of the vertical energy component to the sum of the horizontal and vertical energy components, and wherein the second weighting factor is based at least partially upon the ratio of the horizontal energy component to the sum of the horizontal and vertical energy components; and

summing the weighted vertical and horizontal components to determine the interpolated green value for the current non-green pixel;

interpolating a red value for each blue and green pixel of the raw image pattern to obtain a full set of red values for the image data;

interpolating a blue value for each red and green pixel of the raw image pattern to obtain a full set of blue values for the image data; and

using the full sets of green, red, and blue values to generate a full color image.

2. The method of claim 1, wherein determining the horizontal and vertical energy components is based upon an edge adaptive filter defined as a set of pixels forming a 5×

5 array with the current non-green pixel being at the center of the 5×

5 array, and wherein the 5×

5 array comprises five rows and five columns including a first row defining an upper edge of the 5×

5 array, a second row directly below the first row, a third row directly below the second row and containing the current non-green pixel, a fourth row directly below the third row, a fifth row directly below the fourth row and defining a lower edge of the 5×

5 array, a first column defining a left edge of the 5×

5 array, a second column directly adjacent to the right of the first column, a third column directly adjacent to the right of the second column and containing the current non-green pixel, a fourth column directly adjacent to the right of the third column, and a fifth column directly adjacent to the right of the fourth column and defining a right edge of the 5×

5 array.

3. The method of claim 2, wherein determining the horizontal energy component comprises:

applying a set of coefficients comprising first, second, third, fourth, and fifth coefficients to each of the first, second, third, fourth, and fifth columns, respectively, of the 5×

5 array;

determining a first value as the absolute sum of the sum of the values of the pixels common to the first column and the second, third, and fourth rows of the 5×

5 array multiplied by the first coefficient, the sum of the values of the pixels common to the third column and the second, third, and fourth rows of the 5×

5 array multiplied by the third coefficient, and the sum of the values of the pixels common to the fifth column and the second, third, and fourth rows of the 5×

5 array multiplied by the fifth coefficient;

determining a second value as the absolute sum of the sum of the values of the pixels common to the second column and the second, third, and fourth rows of the 5×

5 array multiplied by the second coefficient and the sum of the value of the pixels common to the fourth column and the second, third, and fourth rows of the 5×

5 array multiplied by the fourth coefficient; and

summing the first value and the second values.

4. The method of claim 2, wherein determining the vertical energy component comprises:

applying a set of coefficients comprising first, second, third, fourth, and fifth coefficients to each of the first, second, third, fourth, and fifth rows, respectively, of the 5×

5 array;

determining a first value as the absolute sum of the sum of the values of the pixels common to the first row and the second, third, and fourth columns of the 5×

5 array multiplied by the first coefficient, the sum of the values of the pixels common to the third row and the second, third, and fourth columns of the 5×

5 array multiplied by the third coefficient, and the sum of the values of the pixels common to the fifth row and the second, third, and fourth columns of the 5×

5 array multiplied by the fifth coefficient;

determining a second value as the absolute sum of the sum of the values of the pixels common to the second row and the second, third, and fourth columns of the 5×

5 array multiplied by the second coefficient and the sum of the value of the pixels common to the fourth row and the second, third, and fourth columns of the 5×

5 array multiplied by the fourth coefficient; and

summing the first value and the second values.

5. The method of claim 2, wherein applying horizontal filtering to determine the horizontal component of the interpolated green value comprises:

applying low pass filtering based upon green pixels neighboring the current non-green pixel in the center row of the 5×

5 array to obtain a horizontal low pass filtered value;

applying high pass filtering based upon the current non-green pixel and the non-green pixels at the left and right edges of the center row of the 5×

5 array to determine a horizontal high pass filtered value; and

summing the horizontal low pass filtered value and the horizontal high pass filtered value to determine the horizontal component of the interpolated green value.

6. The method of claim 5, wherein the horizontal component of the interpolated green value is equivalent to summing the values of the current non-green pixel and each of the two neighboring green pixels of the center row and multiplying the result by a filtering coefficient of 2 to obtain a first value, summing the values of each of the non-green pixels at the left and right edges of the center row and multiplying the result by a filtering coefficient of −

1 to obtain a second value, and dividing the sum of the first and second values by 4.

7. The method of claim 5, wherein applying vertical filtering to determine the vertical component of the interpolated green value comprises:

applying low pass filtering based upon green pixels neighboring the current non-green pixel in the center column of the 5×

5 array to obtain a vertical low pass filtered value;

applying high pass filtering based upon the current non-green pixel and the non-green pixels at the upper and lower edges of the center column of the 5×

5 array to determine a vertical high pass filtered value; and

summing the vertical low pass filtered value and the vertical high pass filtered value to determine the vertical component of the interpolated green value.

8. The method of claim 7, wherein the vertical component of the interpolated green value is equivalent to summing the values of the current non-green pixel and each of the two neighboring green pixels of the center column and multiplying the result by a filtering coefficient of 2 to obtain a first value, summing the values of each of the non-green pixels at the upper and lower edges of the center column and multiplying the result by a filtering coefficient of −

1 to obtain a second value, and dividing the sum of the first and second values by 4.

9. The method of claim 1, wherein interpolating a red value for each blue and green pixel of the raw image pattern to obtain a full set of red values for the image data comprises:

receiving a current non-red pixel;

identifying a set of pixels forming a 3×

3 array with the current non-red pixel being in the center of the 3×

3 array;

determining a low pass filtered component of the interpolated red value for the current non-red pixel by computing the average of the values of the red neighboring pixels within the 3×

3 array;

if the current non-red pixel is green, determining a high pass filtered component by computing the average of the gradients between the value of the current green pixel and each interpolated green value co-located at the positions of the red neighboring pixels;

if the current non-red pixel is blue, determining the high pass filtered component by computing the average of the gradients between an interpolated green value corresponding to the position of the current blue pixel and each interpolated green value co-located at the positions of the red neighboring pixels; and

summing the low pass filtered component and high pass filtered component to determine the interpolated red value for the current non-red pixel.

10. The method of claim 1, wherein interpolating a blue value for each red and green pixel of the raw image pattern to obtain a full set of blue values for the image data comprises:

receiving a current non-blue pixel;

identifying a set of pixels forming a 3×

3 array with the current non-blue pixel being in the center of the 3×

3 array;

determining a low pass filtered component of the interpolated blue value for the current non-blue pixel by computing the average of the values of the blue neighboring pixels within the 3×

3 array;

if the current non-blue pixel is green, determining a high pass filtered component by computing the average of the gradients between the value of the current green pixel and each interpolated green value co-located at the positions of the blue neighboring pixels;

if the current non-blue pixel is red, determining the high pass filtered component by computing the average of the gradients between an interpolated green value corresponding to the position of the current red pixel and each interpolated green value co-located at the positions of the blue neighboring pixels; and

summing the low pass filtered component and high pass filtered component to determine the interpolated blue value for the current non-blue pixel.

11. The method of claim 1, wherein applying the first weighting factor to the horizontal component and the second weighting factor to the vertical component comprises quantizing the first weighting factor and the second weighting factor.

12. An image signal processing circuit, comprising:

an interface configured to receive a raw image pattern based upon image data acquired using an image sensor having a color filter array; and

an image processing pipeline configured to perform a set of processing operations on the image data, wherein one of the set of processing operations is a demosaicing operation, and wherein the image processing pipeline comprises demosaicing logic configured to;

interpolate a green value for each missing green sample in the image pattern to obtain a full set of green values for the image data, wherein the interpolation the missing green samples comprises;

determining horizontal and vertical components of an interpolated green value for a current non-green pixel;

applying edge adaptive filtering to a set of pixels neighboring the current non-green pixel to determine one or more weighting coefficients; and

applying the one or more weighting coefficients to the horizontal and vertical components and summing the weighted horizontal and vertical components to determine the interpolated green value;

interpolate a red value for each missing red sample in the image pattern to obtain a full set of red values for the image data, wherein the interpolation of the missing red samples is based at least partially upon color differences calculated using interpolated green values co-located with red pixels neighboring a current non-red pixel within a 3×

3 pixel array in which the current non-red pixel is located at the center; and

interpolate a blue value for each missing blue sample in the image pattern to obtain a full set of blue values for the image data, wherein the interpolation of the missing blue samples is based at least partially upon color differences calculated using interpolated green values co-located with blue pixels neighboring a current non-blue pixel within a 3×

3 pixel array in which the current non-blue pixel is located at the center.

13. The image processing circuit of claim 12, wherein the horizontal component of the interpolated green value for the current non-green pixel is determined by:

identifying a horizontal row of five adjacent pixels with the current non-green pixel being in the center of the horizontal row;

determining a low pass filtering component by calculating the average of the values of green neighboring pixels directly adjacent to the current non-green pixel within the horizontal row;

if the current non-green pixel is a red pixel, determining a high pass filtering component by calculating the average of the gradients between the current red pixel and interpolated red values co-located with the green neighboring pixels;

if the current non-green pixel is a blue pixel, determining the high pass filtering component by calculating the average of the gradients between the current blue pixel and interpolated blue values co-located with the green neighboring pixels; and

summing the low pass filtered component and the high pass filtered component to determine the horizontal component of the interpolated green value.

14. The image processing circuit of claim 13, wherein the interpolated red values or the interpolated blue values co-located with the neighboring green pixels are determined by linear interpolation of the value of the current non-red pixel and the values of each of the left-most and right-most pixels in the horizontal row.

15. The image processing circuit of claim 14, wherein the vertical component of the interpolated green value for the current non-green pixel is determined by:

identifying a vertical column of five adjacent pixels with the current non-green pixel being in the center of the vertical column;

determining a low pass filtering component by calculating the average of the values of green neighboring pixels directly adjacent to the current non-green pixel within the vertical column;

if the current non-green pixel is a red pixel, determining a high pass filtering component by calculating the average of the gradients between the current red pixel and interpolated red values co-located with the green neighboring pixels;

if the current non-green pixel is a blue pixel, determining the high pass filtering component by calculating the average of the gradients between the current blue pixel and interpolated blue values co-located with the green neighboring pixels; and

summing the low pass filtered component and the high pass filtered component to determine the vertical component of the interpolated green value.

16. The image processing circuit of claim 15, wherein the interpolated red values or the interpolated blue values co-located with the neighboring green pixels are determined by linear interpolation of the value of the current non-red pixel and the values of each of the upper-most and lower-most pixels in the vertical column.

17. The image signal processing circuit of claim 12, wherein the image processing pipeline comprises at least one of noise reduction logic configured to reduce noise in the raw image pattern, lens shading correction logic configured to apply corrective lens shading gains to the image pattern, or gain/offset logic configured to apply auto-white balance to the image pattern, wherein the noise reduction operation, the lens shading correction operation, and the auto-white balance operation occur before the demosaicing operation.

18. The image signal processing circuit of claim 12, wherein the demosaicing logic is configured to reconstruct a full color image based upon the interpolated full sets of green, red, and blue values.

19. The image signal processing circuit of claim 12, wherein the interface comprises a Standard Mobile Imaging Architecture (SMIA) interface or a Mobile Industry Processor Interface (MIPI), or some combination thereof.

20. The image signal processing circuit of claim 12, wherein the color filter array comprises a Bayer color pattern.

21. A system, comprising:

an imaging device comprising an image sensor, wherein the image sensor comprises a color filter array defining a color pattern comprising a first-color component, a second-color component, and a third-color component, wherein the color pattern is arranged such that the distribution of the first-color component within the color pattern is twice as great as the distribution of each of the second-color component and the third-color component; and

image processing circuitry configured to process an image pattern acquired using the image sensor, wherein the image processing circuitry comprises;

demosaicing logic configured to interpolate a first-color value for each missing first-color sample in the image pattern in order to obtain a full set of first-color values for the image data, interpolate a second-color value for each missing second-color sample in the image pattern in order to obtain a full set of second-color values for the image data, and interpolate a third-color value for each missing third-color sample in the image pattern in order to obtain a full set of third-color values for the image data;

wherein the interpolation of the missing first-color samples comprises;

for a current non-first-color pixel, determining horizontal and vertical components of the interpolated first-color value corresponding to the current non-first-color pixel;

applying edge adaptive filtering to a 5×

5 array of pixels to determine horizontal and vertical energy components corresponding to the current non-first-color pixel, wherein the current non-first-color pixel is located at the center of the 5×

5 array;

weighting the horizontal component using a ratio of the vertical energy component to the sum of the vertical and horizontal energy components and weighting the vertical component using a ratio of the horizontal energy component to the sum of the vertical and horizontal energy components; and

determining the interpolated first-color value for the current non-first-color pixel by summing the weighted horizontal and vertical components.

22. The system of claim 21, wherein the demosaicing logic is configured to disable the weighting of the horizontal and vertical components by setting the values of the horizontal and vertical energy components both equal to 1.

23. The system of claim 21, wherein the color filter array comprises a Bayer color filter array, and wherein the first-color component, the second-color component, and the third-color component correspond to green, red, and blue, respectively.

24. The system of claim 21, wherein the image processing circuitry is configured to reconstruct a full color image based upon the full sets of the first-color, second-color, and third-color values.

25. The system of claim 21, comprising an electronic device, and wherein the electronic device comprises one of a laptop computer, a desktop computer, a digital camera, or a handheld portable device, or some combination thereof.

26. A method for processing image data, comprising:

using an image processor;

receiving a raw image pattern based upon image data acquired using an image sensor, the raw image pattern comprising a plurality of green, red, and blue pixels arranged in accordance with a color filter array; and

interpolating a green value for each non-green pixel in the image pattern to obtain a full set of green values for the image data, wherein interpolating the green value comprises;

using edge adaptive filtering to determine a horizontal energy component, a vertical energy component, a diagonal-positive energy component, and a diagonal negative energy component for a current non-green pixel;

determining a horizontal component, a vertical component, a diagonal-positive component, and a diagonal-negative component of an interpolated green value that correspond to the horizontal energy component, the vertical energy component, the diagonal-positive energy component, and the diagonal negative energy component, respectively, for the current non-green pixel; and

applying the one or more weighting coefficients to at least two of the horizontal, vertical, diagonal-positive, and diagonal-negative components; and

summing the weighted at least two components to determine the interpolated green value for the current non-green pixel;

29. The method of claim 26, wherein the color filter array comprises a Bayer color pattern.

30. The method of claim 26, wherein applying the one or more weighting coefficients to at least two of the horizontal, vertical, diagonal-positive, and diagonal-negative components comprises:

selecting the two highest energy components of the horizontal, vertical, diagonal-positive, or diagonal-negative energy components;

determining weighting coefficients corresponding to each of the selected two highest energy components; and

applying the determined weighting coefficients to the two of the horizontal, vertical, a diagonal-positive component, and a diagonal-negative components that correspond to the two selected energy components.

27. The method of claim 27, comprising:

interpolating a red value for each blue and green pixel of the raw image pattern to obtain a full set of red values for the image data, wherein interpolating the red value is based at least partially on the average of the ratios of the values of at least two red pixels neighboring a current non-red pixel to at least two co-located interpolated green values within a 3×

3 pixel array in which the current non-red pixel is located at the center; and

interpolating a blue value for each red and green pixel of the raw image pattern to obtain a full set of blue values for the image data, wherein interpolating the blue value is based at least partially on the average of the ratios of the values of at least two blue pixels neighboring a current non-blue pixel to at least two co-located interpolated green values within a 3×

3 pixel array in which the current non-blue pixel is located at the center.

28. The method of claim 27, wherein interpolating the red value for a non-red pixel comprises:

if the non-red pixel is green, determining the average of the ratio of the value of a first red neighboring pixel within the 3×

3 array to a first co-located interpolated green value and the ratio of the value of a second red neighboring pixel within the 3×

3 array to a second co-located interpolated green value, and multiplying the determined average by the value of the current green pixel to determine the interpolated red value for the current green pixel; and

if the non-red pixel is blue, determining the average of the ratios of the values of first, second, third, and fourth red neighboring pixels within the 3×

3 array to respective co-located interpolated green values, and multiplying the determined average by an interpolated green value co-located with the value of the current blue pixel to determine the interpolated red value for the current blue pixel; and

wherein interpolating the blue value for a non-blue pixel comprises;

if the non-blue pixel is green, determining the average of the ratio of the value of a first blue neighboring pixel within the 3×

3 array to a first co-located interpolated green value and the ratio of the value of a second blue neighboring pixel within the 3×

3 array to a second co-located interpolated green value, and multiplying the determined average by the value of the current green pixel to determine the interpolated blue value for the current green pixel; and

if the non-blue pixel is red, determining the average of the ratios of the values of first, second, third, and fourth blue neighboring pixels within the 3×

3 array to respective co-located interpolated green values, and multiplying the determined average by an interpolated green value co-located with the value of the current red pixel to determine the interpolated blue value for the current red pixel.