System and method for sharpening image data

• US 8,294,781 B2
• Filed: 10/20/2009
• Issued: 10/23/2012
• Est. Priority Date: 10/20/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Cote, Guy, Frederiksen, Jeffrey E.
• Primary Examiner(s): NGUYEN, LUONG TRUNG
• Application Number:
  US12/582,390
• Publication Number:
  US 20110090370A1
• Time in Patent Office: 1,099 Days
• Field of Search: None
• US Class Current: 348/222.1
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
• CPC Class Codes: H04N 25/63 applied to dark current
• CPC Class Codes: H04N 25/68 applied to defects
• CPC Class Codes: H04N 25/683 by defect estimation performed on the scene signal, e.g. real time or on the fly detection
• CPC Class Codes: H04N 5/208 for compensating for attenuation of high frequency components, e.g. crispening, aperture distortion correction

Abstract

Various techniques relating to image sharpening are provided. In one embodiment, a luminance image is obtained based upon image data acquired by an image sensor. A multi-scale unsharp mask, which may include at least two Gaussian filters of difference radii, is applied to the luminance image to determine a plurality of unsharp values. Each of the unsharp values may be compared to a corresponding threshold and, for the unsharp values that exceed their respective thresholds, the unsharp value is multiplied by a corresponding gain and added to a base image, which may be selected as the luminance image or the output of one of the Gaussian filters. Each gained unsharp value may be summed with the base image to produce a final sharpened output. In some embodiments, an attenuated gain may be applied to unsharp values that do not exceed their respective thresholds.

Claims

1. A method for processing an image, comprising:

using an image processor;

obtaining a luma image based upon image data acquired using an image sensor;

for each luma pixel of the luma image, applying a multi-scale unsharp mask filter to a current luma pixel to determine one or more unsharp values, wherein each of the one or more unsharp values is associated with a respective gain and a respective threshold;

determining a base value, wherein the base value is selected as either a value of the current luma pixel or as a value of an output of the multi-scale unsharp mask filter based at least partially upon a noise variance of the image data;

comparing each of the one or more unsharp values to the respective threshold;

for each of the one or more unsharp values that is greater than the respective threshold, applying the respective gain to determine corresponding gained unsharp values; and

adding each gained unsharp value to the base value to produce a sharpened pixel corresponding to the current luma pixel.

2. The method of claim 1, wherein the value of the current luma pixel is selected as the base value if the noise variance is substantially lower than a signal variance of the image data, and wherein the value of an output of the multi-scale unsharp mask filter is selected as the base value otherwise.

3. The method of claim 1, wherein the multi-scale unsharp mask filter comprises at least two low pass filters, including a first low pass filter and a second low pass filter, wherein the first and second low pass filters have different scale sizes.

4. The method of claim 3, wherein the first low pass filter comprises a Gaussian filter having a first radius, and wherein the second low pass filter comprises a Gaussian filter having a second radius, wherein the second radius is greater than the first radius.

5. The method of claim 1, wherein obtaining the luma image comprises:

demosaicing the image data acquired by the image sensor to obtain a full color RGB image; and

converting the full color RGB image to the YCbCr color space to obtain a YCbCr image, wherein the luma image is defined by the luma components of the YCbCr image.

6. A method for applying sharpening to an image, comprising:

obtaining a luma image based upon image data acquired using an image sensor;

for each luma pixel of the luma image, identifying a current luma pixel and applying a multi-scale unsharp mask filter to the current luma pixel to determine one or more unsharp values, wherein each of the one or more unsharp values is associated with a respective gain and a respective threshold;

comparing each of the one or more unsharp values to the respective threshold;

for each of the one or more unsharp values that is greater than the respective threshold, applying the respective gain to determine a first set of corresponding gained unsharp values;

for each of the one or more unsharp values that is less than its respective threshold, applying an attenuated gain based upon its respective gain to determine a second set of corresponding gained unsharp values; and

adding the gained unsharp values from the first and second sets to a base value to produce a sharpened pixel corresponding to the current luma pixel.

7. The method of claim 6, wherein the base value is selected as either the value of the current luma pixel or as a value of an output of the multi-scale unsharp mask filter, the selection being based at least partially upon the noise variance of the image data.

8. The method of claim 6, wherein, for each of the one or more unsharp values that is less than the respective threshold, the attenuated gain is determined by multiplying the unsharp value with the ratio of the respective gain to its respective threshold.

9. The method of claim 6, wherein the multi-scale unsharp mask filter comprises a first Gaussian low pass filter having a first radius, and a second Gaussian low pass filter having a second radius, wherein the second radius is greater than the first radius.

10. The method of claim 9, wherein the first Gaussian filter comprises a 3×

3 Gaussian filter, and wherein the second Gaussian filter comprises a 5×

5 Gaussian filter.

11. The method of claim 9, wherein the one or more unsharp values comprises:

a first unsharp value determined by subtracting an output of the second Gaussian filter from an output of the first Gaussian filter;

a second unsharp value determined by subtracting the output of the second Gaussian filter from the current luma pixel value; and

a third unsharp value determined by subtracting the output of the first Gaussian filter from the current luma pixel value.

12. The method of claim 11, comprising applying an edge enhancement operation to the sharpened pixel, wherein applying the edge enhancement operation comprises:

applying an edge detection filter to the current luma pixel to determine a directional gradient;

selecting one of the directional gradient value or the first unsharp value, and comparing the selected value to an edge enhancement threshold; and

if the selected value is greater than the edge enhancement threshold, applying an edge enhancement gain to the directional gradient and adding the gained directional gradient to the sharpened pixel.

13. The method of claim 12, wherein the edge detection filter comprises a Sobel filter.

14. The method of claim 12, comprising, if the selected value is not greater than the edge enhancement threshold, applying an attenuated edge enhancement gain to the directional gradient and adding the attenuated gained directional gradient to the sharpened pixel;

wherein the attenuated edge enhancement gain is determined by multiplying the directional gradient with the ratio of the edge enhancement gain to the edge enhancement threshold.

15. The method of claim 12, wherein the selection of the directional gradient value or the first unsharp value is based at least partially upon the noise variance of the image data.

16. An image processing method, comprising:

using an image processor;

obtaining a luma image based upon image data acquired using an image sensor;

for each luma pixel of the luma image, identifying a current luma pixel and applying a multi-scale unsharp mask filter to the current luma pixel, wherein the multi-scale unsharp mask filter comprises a first low pass filter and a second low pass filter, wherein the second low pass filter is larger in size compared to the first low pass filter;

determining a set of unsharp values comprising a first unsharp value determined by subtracting a second low pass filter output from a first low pass filter output, a second unsharp value determined by subtracting the second low pass filter output from a current luma pixel value, and a third unsharp value determined by subtracting the first low pass filter output from the current luma pixel value;

performing a first comparison comprising selecting the first unsharp value, comparing a first threshold to the selected first unsharp value selected during the first comparison and, if the selected first unsharp value is greater than the first threshold, applying a first gain to the first unsharp value to compute a first sharpening component;

performing a second comparison comprising selecting one of either the second unsharp value or the first unsharp value, comparing a second threshold to the selected second or first unsharp value selected during the second comparison, and applying a second gain to the second unsharp value to compute a second sharpening component if the selected unsharp value is greater than the second threshold;

performing a third comparison comprising selecting one of either the third unsharp value or the first unsharp value, comparing a third threshold to the selected third or first unsharp value selected during the third comparison, and applying a third gain to the third unsharp value to compute a third sharpening component if the selected third or first unsharp value is greater than the third threshold; and

for each of the first, second, or third comparisons in which a respective selected unsharp value is greater than the respective first, second, or third thresholds, adding the respective first, second, or third sharpening components to a base value.

17. The method of claim 16, comprising:

if the selected first unsharp value is not greater than the first threshold during the first comparison, applying a portion of the first gain to the first unsharp value to compute a first attenuated sharpening component;

if the selected second or first unsharp value selected during the second comparison is not greater than the second threshold during the second comparison, applying a portion of the second gain to the second unsharp value to compute a second attenuated sharpening component;

if the selected third or first unsharp value selected during the third comparison is not greater than the third threshold during the third comparison, applying a portion of the third gain to the third unsharp value to compute a third attenuated sharpening component; and

for each of the first, second, or third comparisons in which a respective selected unsharp value is not greater than the respective first, second, or third thresholds, adding the respective first, second, or third attenuated sharpening components to the base value.

18. The method of claim 16, wherein the selection of either the first unsharp value or the second unsharp value during the second comparison and the selection of either the first unsharp value or the third unsharp value during the third comparison is based at least partially upon noise variance of the image data.

19. The method of claim 16, wherein the first and second low pass filters each comprise a Gaussian filters.

20. The method of claim 16, wherein the base value is selected as being either the value of the current luma pixel, the output of the first low pass filter, or the output of the second low pass filter based at least partially upon the noise variance of the image data.

21. An image processing circuit comprising:

an interface configured to receive image data from an image sensor; and

image sharpening logic configured to sharpen a luma image derived from the image data, the image sharpening logic comprising;

an input configured to receive luma pixels corresponding to the luma image;

a multi-scale unsharp mask filter comprising a first low pass filter having a first size and a second low pass filter having a second size, wherein the multi-scale unsharp mask filter is configured to apply the first and the second low pass filters to a current luma pixel, and wherein the image sharpening logic is configured to determine a first unsharp value by subtracting a second low pass filter output from a first low pass filter output, a second unsharp value by subtracting the second low pass filter output from a current luma pixel value, and a third unsharp value by subtracting the first low pass filter output from the current luma pixel value; and

adaptive coring logic comprising;

a first selection circuit configured to receive the first unsharp value and the second unsharp value and to output either the first unsharp value or the second unsharp value based upon the state of a first control signal;

a second selection circuit configured to receive the first unsharp value and the third unsharp value and to output either the first unsharp value or the third unsharp value based upon the state of a second control signal;

a first comparator block configured to compare a first threshold against the first unsharp value;

a second comparator block configured to compare a second threshold against the output of the first selection circuit; and

a third comparator block configured to compare a third threshold against the output of the second selection circuit;

wherein a sharpened luma pixel corresponding to the current luma pixel is determined by adding the first unsharp value multiplied by a first gain to a base value if the output of the first comparator block indicates that the first unsharp value is greater than the first threshold, adding the second unsharp value multiplied by a second gain to the base value if the output of the second comparator block indicates that the output of the first selection circuit is greater than the second threshold, and adding the third unsharp value multiplied by a third gain to the base value if the output of the third comparator block indicates that the output of the second selection circuit is greater than the third threshold.

22. The image processing circuit of claim 21, wherein the states of the first and second control signals are determined based at least partially upon the noise variance of the image data.

23. The image processing circuit of claim 22, wherein the first selection circuit is configured to output the second unsharp value if the noise variance is substantially lower than the signal variance of the image data and to output the first unsharp value otherwise;

and

wherein the second selection circuit is configured to output the third unsharp value if the noise variance is substantially lower than the signal variance of the image data and to output the first unsharp value otherwise.

24. The image processing circuit of claim 21, comprising a third selection circuit configured receive the current luma pixel, the output of the first low pass filter, and the output of the second low pass filter, and to output either the current luma pixel, the output of the first low pass filter, or the output of the second low pass filter based upon the state of a third control signal, wherein the base value is equal to the output of the third selection circuit.

25. The image processing circuit of claim 21, comprising edge enhancement logic configured to enhance edges in the sharpened luma image using an edge detection filter.

26. The image processing circuit of claim 21, comprising chroma suppression logic configured to reduce the appearance of chroma near edges in the sharpened luma image by applying a chroma attenuation factor to each chroma pixel associated with the sharpened luma image, wherein the amount of attenuation applied is generally inversely proportional to the value of a respective luma pixel in the sharpened luma image.