SPATIAL FILTERING FOR IMAGE SIGNAL PROCESSING

• US 20120081553A1
• Filed: 09/30/2010
• Published: 04/05/2012
• Est. Priority Date: 09/30/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Côté, Guy, Arnold, Vaughn Todd, Chawla, Sumit
• Granted Patent: US 8,471,932 B2
• Time in Patent Office: Days
• US Class Current: 348/207.1
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 2207/20028 Bilateral filtering
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

Systems and methods are disclosed for applying spatial filtering to raw image data. In one embodiment, a spatial filter may identify an n×n block of pixels from the current image frame, the n×n block including a plurality of neighboring pixels being centered about a current input pixel and being of the same color component as the current input pixel. The spatial filter may include a plurality of filter taps, with one filter tap corresponding to each of the pixels within the n×n block. A set of filtering coefficients for each filter tap, which may be based on a Gaussian function, may be determined. A pixel difference value between the current input pixel and each of the plurality of neighboring pixels in the n×n block are determined, and the pixel differences may be used to determine an attenuation factor for each filter tap. In one embodiment, the attenuation factor is based upon the pixel difference value between the raw input pixel and the neighboring pixel at the filter tap and brightness of the current input pixel. An attenuated set of filtering coefficients may be obtained thereafter by applying the attenuation factors to their respective filtering taps. Subsequently, the attenuated filtering coefficients may be applied to obtain a set of filtered pixel values. By normalizing the sum of the filtered pixel values using the sum of the attenuated filtering coefficients, a spatially filtered output value corresponding to the current input pixel (e.g., located at the center of the n×n block) may be determined.

Claims

1. A method comprising

receiving an image frame comprising raw pixels, wherein the image frame is acquired using an image sensor, and wherein the raw pixels comprise multi-color raw pixel data;

applying spatial noise filtering to each raw pixel of the image frame using a spatial noise filter, wherein applying spatial noise filtering comprises;

identifying a first set of pixels from the image frame, the set of pixels comprising a current input pixel and a plurality of same-colored neighboring pixels, wherein the current input pixel has a first spatial location within the image frame and a pixel value, and wherein the neighboring pixels are spatially located about the current input pixel;

identifying a set of filtering coefficients for each filter tap of the spatial noise filter, wherein the set of filtering coefficients comprise a respective filtering coefficient for the each of the first set of pixels;

determining a pixel difference value between the current input pixel and each neighboring pixel of the first set of pixels;

determining an attenuation factor for each filtering coefficient of the spatial noise filter using the pixel difference values;

applying a respective attenuation factor to each filtering coefficient to determine a set of attenuated filtering coefficients;

applying the attenuated filtering coefficients to the first set of pixels to obtain a set of filtered pixel values; and

providing a spatially filtered output value corresponding to the current input pixel by normalizing the sum of the filtered pixel values using the sum of the set of attenuated filtering coefficients.

2. The method of claim 1, wherein applying spatial noise filtering comprises operating the spatial noise filter in one of a bilateral filtering mode or a non-local means filtering mode.

3. The method of claim 2, wherein if the spatial noise filter is operating in the bilateral filtering mode, determining a pixel difference value between the current input pixel and each neighboring pixel of the first set of pixels comprises:

for each neighboring pixel, determining the absolute value of the difference between the pixel value of the current pixel and a pixel value of the neighboring pixel.

4. The method of claim 2, wherein if the spatial noise filter is operating in the non-local means filtering mode, determining a pixel difference value between the current input pixel and each neighboring pixel of the first set of pixels comprises:

identifying a first subset of the neighboring pixels centered about the current input pixel; and

for each neighboring pixel, identifying a second subset of neighboring pixels centered about the neighboring pixel, determining a sum of the absolute value of the differences between the pixels of the first subset and the second subset, and normalizing the sum of the absolute value of the differences based on a number of pixels equal to the number of pixels of the first subset and the current input pixel.

5. The method of claim 1, wherein determining an attenuation factor for each filtering coefficient of the spatial noise filter using the pixel difference values comprises:

determining an inverse noise standard deviation value based upon the pixel value of the current input pixel;

determining the radial distance of the current input pixel relative to a center of the image frame;

using the radial distance to select a radial gain;

applying the radial gain to the inverse noise standard deviation value to obtain a gained inverse noise standard deviation value; and

for each neighboring pixel, determining a respective attenuation factor based on the gained inverse noise standard deviation value and the pixel difference value between the current input pixel and the neighboring pixel, and associating the attenuation factor with a filter tap of the spatial noise filter that corresponds to the neighboring pixel.

6. The method of claim 1, wherein determining an attenuation factor for each filtering coefficient of the spatial noise filter using the pixel difference values comprises:

receiving a brightness value parameter corresponding the current input pixel;

receiving a pixel difference value corresponding to the difference between the current input pixel and a neighboring pixel located at a current filter tap having a filtering coefficient;

evaluating the brightness value parameter to select a brightness interval in which the current input pixel belongs, wherein the selected brightness interval includes a lower and upper brightness level;

determining a first lower pixel difference level and a first upper pixel difference level based on the lower brightness level;

determining a second lower pixel difference level and a second upper pixel difference level based on the upper brightness level;

determining a first attenuation factor from an attenuation factor look-up table based upon the lower brightness level, the first lower pixel difference level, and the first upper pixel difference level;

determining a second attenuation factor from the attenuation factor look-up table based upon the upper brightness level, the second lower pixel difference level, and the second upper pixel difference level; and

interpolating the first and second attenuation factors to obtain the attenuation factor for the current filter tap.

7. The method of claim 6, wherein the brightness value parameter comprises an average brightness of same-colored pixels in a 3×

3 block centered at the current input pixel.

8. The method of claim 6, wherein the attenuation factor look-up table provides attenuation factors having a range of between 0.0 to 1.0.

9. The method of claim 6, wherein the attenuation factor look-up table provides factors having a range of between 0 and a value greater than 1.0, and wherein if a factor of greater than 1.0 is selected, image sharpening is implemented at the current filter tap.

10. The method of claim 1, wherein the multi-color raw pixel data comprises Bayer raw pixel data having a red component, a first green component, a second green component, and a blue component.

11. The method of claim 1, wherein the first set of pixels comprises an n×

n block of same-colored pixels, wherein n≧

3.

12. An image signal processing system comprising

a spatial noise filter for spatially filtering multi-color raw image data acquired using an image sensor, wherein the spatial noise filter is configured to:

receive a current pixel from a current frame of raw image data, the current pixel being of a first color and having a first value and a spatial location within the current frame;

identify a group of neighboring pixels of the first color that are centered about the current pixel, wherein the identified group of neighboring pixels and the current pixel form an n×

n block of pixels of the first color, wherein each pixel of the n×

n block is associated with a respective filter tap of the spatial noise filter;

determine an absolute difference value between the current pixel and each neighboring pixel of the n×

n block;

determine an inverse noise standard deviation parameter for the current pixel based on a brightness level corresponding to the first value and a radial gain based on the distance between the spatial location of the current pixel and a center point of the current frame;

for each neighboring pixel of the n×

n block, determine an attenuation factor based on the inverse noise standard deviation parameter for the current pixel and the absolute difference value between the current pixel and the neighboring pixel, apply the attenuation factor to a filter coefficient at the filter tap associated with the neighboring pixel to obtain an attenuated filter coefficient at the filter tap;

apply the attenuated filter coefficients to the pixels within the n×

n block to obtain a filtered pixel value for each pixel of the n×

n block;

obtain a sum of the attenuated filter coefficients at each filter tap and a sum of the filtered pixel values; and

providing a spatially filtered output value by dividing the sum of the filtered pixel values by the sum of the attenuated filter coefficients.

13. The image signal processing system of claim 12, wherein the spatial noise filter is configured to determine the inverse noise standard deviation parameter by:

providing a plurality of brightness intervals, each interval having an upper and lower brightness value;

evaluating the first value to select a brightness interval in which the current pixel belongs;

determine a first inverse noise standard deviation value corresponding to the upper brightness value of the selected brightness interval;

determine a second inverse noise standard deviation value corresponding to the lower brightness value of the selected brightness interval;

interpolating the first and second inverse noise standard deviation values to determine a third inverse noise standard deviation value; and

applying the radial gain to the third inverse noise standard deviation value to obtain the inverse noise standard deviation parameter.

14. The image signal processing system of claim 12, wherein the n×

n block comprises at least a 7×

7 block of pixels of the first color.

15. The image signal processing system of claim 12, the image processing system comprises a temporal filter configured to apply temporal filtering to the multi-color raw image data.

16. The image signal processing system of claim 15, wherein the temporal filter is applied before the spatial noise filter.

17. The image signal processing system of claim 16, comprising defective pixel correction logic, wherein the defective pixel correction logic is configured to provide for at least one of static defect correction, dynamic defect correction, and speckle correction prior to spatial filtering by the spatial noise filter.

18. The image signal processing system of claim 17, wherein the temporal filter, the defective pixel correction logic, and the spatial noise filter form a front-end pixel processing unit of the image signal processing system.

19. The image signal processing system of claim 18, wherein the front-end pixel processing unit comprises a binning compensation filter located downstream from the spatial noise filter.

20. A spatial filtering method comprising:

receiving image frames comprising multi-colored raw pixels, wherein the image frames are acquired using an image sensor;

using a spatial noise filter to apply three-dimensional spatial noise filter an input pixel from a current image frame from a first time, wherein applying three-dimensional spatial noise filtering comprises;

determining a first spatially filtered value corresponding to the input pixel using pixels from the current image frame from the first time;

determining a second spatially filtered value corresponding to the input pixel using pixels from a previous image frame from the second time; and

determining a spatially filtered output value for the input pixel based upon an average of the first and second spatially filtered values.

21. The spatial filtering method of claim 20, wherein determining the first spatially filtered value comprises:

identifying in the current image frame a first group of same-colored neighboring pixels centered about the input pixel and forming a first n×

n block of pixels, wherein each pixel of the first n×

n block is associated with a respective filter tap of the spatial noise filter;

determining an absolute difference value between the current pixel and each neighboring pixel of the first n×

n block;

using the absolute difference values between the current pixel and each neighboring pixel of the first n×

n block, determining a first set of attenuation factors corresponding to filtering coefficients associated with the filter taps of the spatial noise filter;

applying the first set of attenuation factors to the filtering coefficient to determine a first set of attenuated filtering coefficients;

applying the first set of attenuated filtering coefficients to the pixels of the first n×

n block to determine a first set of filtered pixel values; and

determining the first spatially filtered value by normalizing the sum of each of the first set of filtered pixel values by the sum of each of the first set of attenuated filtering coefficients; and

wherein determining the second spatially filtered value comprises;

identifying in the previous image frame a second group of same-colored neighboring pixels co-located with the first group of same-colored neighboring pixels identified from the current image frame, wherein the input pixel and the second group of same-colored neighboring pixels form a second n×

n block of pixels, wherein each pixel of the second n×

n block is associated with a respective one of the filter taps of the spatial noise filter;

determining an absolute difference value between the current pixel and each neighboring pixel of the second n×

n block;

using the absolute difference values between the current pixel and each neighboring pixel of the second n×

n block, determining a second set of attenuation factors corresponding to filtering coefficients associated with the filter taps of the spatial noise filter;

applying the second set of attenuation factors to the filtering coefficient to determine a second set of attenuated filtering coefficients;

applying the second set of attenuated filtering coefficients to the pixels of the second n×

n block to determine a second set of filtered pixel values; and

determining the second spatially filtered value by normalizing the sum of each of the second set of filtered pixel values by the sum of each of the second set of attenuated filtering coefficients.

22. The method of claim 21, wherein determining the first set of attenuation factors comprises, for each neighboring pixel of the first n×

n block of pixels;

using a brightness value of the input pixel to determine an inverse noise standard deviation parameter;

applying to the inverse noise standard deviation parameter, a radial gain determined based upon the radial distance of the input pixel from the center of the current frame; and

using the gained inverse noise standard deviation parameter and a delta value equal to the absolute difference value between the input pixel and the current neighboring pixel to derive an attenuation factor corresponding to the filter tap associated with the current neighboring pixel.

23. The method of claim 22, wherein the attenuation factor is derived using an exponential function, with the gained inverse noise standard deviation parameter and the delta value being exponential components of the exponential function.

24. The method of claim 22, wherein determining the radial gain comprises:

providing a plurality of radial distance intervals between the center of the current frame and a maximum radius value, each radial distance interval having an upper and lower radius;

evaluating the radial distance of the input pixel from the center of the current frame to select a radial distance interval in which the radial distance;

determine a first radial gain value corresponding to the upper radius of the selected radial distance interval;

determine a second radial gain value corresponding to the lower radius of the selected radial distance interval; and

interpolating the first and second radial gain values to determine the radial gain.

25. The method of claim 20, wherein determining a spatially filtered output value for the input pixel based upon an average of the first and second spatially filtered values comprises weighting the first spatially filtered value by a greater amount than the second spatially filtered value.

26. An electronic device comprising:

an image sensor configured to acquire raw image data comprising a plurality of raw pixels representative of an image scene;

an image signal processing sub-system; and

a memory device configured to store the raw image data;

a sensor interface configured to provide the raw pixels from the image sensor at least one of the memory or the image signal processing sub-system, wherein the image signal processing sub-system comprises a front-end processing unit comprising;

a spatial noise filter comprising a plurality of filter taps and configured to apply spatial filtering to a raw input pixel a current image frame by identifying an n×

n block of pixels from the current image frame, the n×

n block including a plurality of neighboring pixels being centered about the raw input pixel and being of the same color component as the raw input pixel, wherein each pixel of the n×

n block corresponds to a respective one of the plurality of filter taps of the spatial noise filter, identifying a set of filtering coefficients for each filter tap of the spatial noise filter, determining a pixel difference value between the raw input pixel and each of the plurality of neighboring pixels in the n×

n block, attenuating each of the filtering components by determining an attenuation factor for each filter tap, wherein the attenuation factor is based at least partially upon the pixel difference value between the raw input pixel and the neighboring pixel at the filter tap, applying a respective attenuation factor to each filtering coefficient of the spatial noise filter to determine a set of attenuated filtering coefficients, applying the attenuated filtering coefficients to the pixels of the n×

n block to obtain a set of filtered pixel values, and outputting a spatially filtered value by normalizing the sum of the filtered pixel values based upon the sum of the set of attenuated filtering coefficients.

27. The electronic device of claim 26, wherein the image sensor comprises a Bayer color filter array, and wherein the raw pixels comprise Bayer pixels.

28. The electronic device of claim 26, wherein the image sensor comprises at least one of a digital camera integrated with the electronic device, an external digital camera coupled to the electronic device via the interface, or some combination thereof.

29. The electronic device of claim 26, wherein the sensor interface comprises a Standard Mobile Imaging Architecture (SMIA) interface, a serial interface, a parallel interface, or some combination thereof.

30. The electronic device of claim 26, comprising at least one of a desktop computer, a laptop computer, a tablet computer, a mobile cellular telephone, a portable media player, or any combination thereof.