Adaptive spatial variant interpolation for image upscaling

• US 9,013,511 B2
• Filed: 08/09/2006
• Issued: 04/21/2015
• Est. Priority Date: 08/09/2006
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Liang, Yi, Lai, King-Chung, El-Maleh, Khaled Helmi
• Primary Examiner(s): Martello, Edward
• Application Number:
  US11/502,233
• Publication Number:
  US 20080036792A1
• Time in Patent Office: 3,177 Days
• Field of Search: 345/660
• US Class Current: 345/660
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403

Abstract

This disclosure describes adaptive spatial variant interpolation (SVI) techniques for image upscaling. In various embodiments, the interpolation techniques described in this disclosure may support low complexity upscaling of image while promoting high image quality, including enhanced sharpness, higher contrast and more accurate interpolation. The interpolation techniques may be applied using generalized finite impulse response (FIR) filters. In some embodiments, the interpolation techniques may be content-adaptive to provide more accurate interpolation while suppressing significant artifacts associated with sharp edges. In addition, the interpolation techniques may be readily applicable to upscaling of color imagery and video, e.g., in both YCbCr (luminance, blue chrominance, red chrominance) and RGB (red, green, blue) formats.

Claims

1. A method comprising:

determining, via a processor, a symmetry (A) of a scaled pixel in an image based on content characteristics of neighboring pixels in the image;

determining, via the processor, an intensity (k) of a spatial variant interpolation filter based on the symmetry (A);

adjusting, via the processor, the spatial variant interpolation filter based on the intensity (k), wherein adjusting the spatial variant interpolation filter includes determining a warped distance (s′

) based on the symmetry (A) of the scaled pixel and the intensity (k) of the filter response, and adjusting the spatial variant interpolation filter based on the warped distance (s′

); and

applying, via the processor, the spatial variant interpolation filter to interpolate the scaled pixel,

wherein determining the intensity (k) of the filter response comprises;

determining whether the symmetry (A) resides within one of a plurality of zones (A₀-A_n); and

setting the intensity (k) to one of a plurality of intensity values (k₀-k_n) associated with the one of the plurality of zones (A₀-A_n) in which the symmetry (A) lies.

2. The method of claim 1, wherein determining the symmetry (A) of the scaled pixel comprises determining the symmetry (A) of the scaled pixel according to the equation:

A=(|f(x_n+1)−

f(x<sub>n−

1</sub>)|−

|f(x_n+2)−

f(x_n)|)>

>

8

where the neighboring pixels comprise pixels denoted by x_n+1, x<sub>n−

1</sub>, x_n+2, and x_n, and f(x_n+1), f(x<sub>n−

1</sub>), f(x_n+2), and f(x_n) denote the value associated with the respective pixels x_n+1,x<sub>n−

1</sub>, x_n+2, and x_n.

3. The method of claim 1, further comprising selectively setting the intensity (k) to equal zero such that the warped distance (s′

) equals a distance (s) from the scaled pixel to one of the neighboring pixels to transform the spatial variant interpolation filter into a spatial invariant interpolation filter, and applying the spatial invariant interpolation filter to interpolate the scaled pixel.

4. The method of claim 1, wherein setting the intensity (k) comprises:

accessing one of a plurality of data structures that store the association of zones (A₀-A_n) to the plurality of intensity values (k₀-k_n) based on the determined symmetry (A) to retrieve one of the plurality of intensity values (k₀-k_n); and

setting the intensity (k) to the retrieved one of the plurality of intensity values (k₀-k_n).

5. The method of claim 4, wherein accessing one of the plurality of data structures comprises accessing one of the plurality of data structures based on one or more of available system resources, and a type of an application requesting the image.

6. The method of claim 1, wherein determining the intensity (k) of the filter response comprises:

determining a quality of the image; and

determining the intensity (k) of the filter response for the scaled pixel based on the determination of the quality of the image.

7. The method of claim 1, wherein determining the warped distance (s′

) comprises determining the warped distance (s′

) according to the equation;

s′

=s−

kAs(s−

1)

where s denotes the distance of the scaled pixel to one of the neighboring pixels, A denotes the symmetry of the scaled pixel, and k denotes the intensity of the filter response for the scaled pixel.

8. The method of claim 1,

wherein determining the warped distance (s′

) comprises determining the warped distance (s′

) for the green (G) channel of the image encoded according to the RGB color format, and

wherein applying the filter comprises applying the filter to the red (R), green (G), and blue (B) channels based on the warped distance (s′

) determined for the green (G) channel of the image to perform spatial variant interpolation to determine the value of the scaled pixel.

9. The method of claim 1,

wherein determining the warped distance (s′

) comprises determining the warped distance (s′

) for the luminance (Y) channel of the image encoded according to the YCbCr color format;

wherein applying the filter comprises applying the filter to the luminance (Y) channel based on the warped distance (s′

) determined for the luminance (Y) channel to perform spatial variant interpolation to determine the value of the scaled pixel.

10. The method of claim 1, wherein applying the filter comprises applying a filter based on the determined warped distance (s′

) to perform spatial variant bicubic interpolation to determine the value of the scaled pixel.

11. The method of claim 10, wherein applying the filter further comprises applying the filter described by the equation:

12. The method of claim 1, further comprising:

generating upscaled image that includes the value of the scaled pixel; and

displaying the upscaled image via a display.

13. A device comprising:

a memory that stores an image; and

a processor that determines a symmetry (A) of a scaled pixel in an image based on content characteristics of neighboring pixels in the image, determines an intensity (k) of an interpolation filter based on the symmetry (A), adjusts the interpolation filter based on the intensity (k), and applies the spatial variant interpolation filter to interpolate the scaled pixel,

wherein the processor that adjusts the interpolation filter based on the intensity (k) determines a warped distance (s′

) based on the symmetry (A) of the scaled pixel and the intensity (k) of the filter response, and adjusts the spatial variant interpolation filter based on the warped distance (s′

), and

wherein the processor determines whether the symmetry (A) resides within one of a plurality of zones (A₀-A_n,), and sets the intensity (k) to one of a plurality of intensity values (k₀-k_n) associated with the one of the plurality of zones (A₀-A_n) in which the symmetry (A) lies.

14. The device of claim 13, wherein the processor determines the symmetry (A) of the scaled pixel according to the equation:

A=(|f(x_n+1)−

f(x<sub>n−

1</sub>)|−

|f(x_n+2)−

f(x_n)|)>

>

8

where the neighboring pixels comprise pixels denoted by x_n+1, x<sub>n−

1</sub>, x_n+2, and x_n, and f(x_n+1), f(x<sub>n−

1</sub>), f(x_n+2), and f(x_n) denote the value associated with the respective pixels x_n+.

15. The device of claim 13, wherein the processor selectively sets the intensity (k) to equal zero such that the warped distance (s′

) equals a distance (s) from the scaled pixel to one of the neighboring pixels to transform the spatial invariant interpolation filter into a spatial invariant interpolation filter, and applies the spatial invariant interpolation filter to interpolate the scaled pixel.

16. The device of claim 13, wherein the memory stores a plurality of data structures that store an association between zones of symmetry (A₀-A_n) and a plurality of intensity values (k₀-k_n), and the processor accesses one of the plurality of data structures that stores the association of zones (A₀-A_n) to the plurality of intensity values (k₀-k_n) based on the determined symmetry (A) to retrieve one of the plurality of intensity values (k₀-k_n), and sets the intensity (k) to the retrieved one of the plurality of intensity values (k₀-k_n).

17. The device of claim 16, wherein the processor accesses one of the plurality of data structures based on one or more of available system resources, and a type of application requesting the image.

18. The device of claim 13, wherein the processor determines quality of the image, and determines the intensity (k) of the filter response for the scaled pixel based on the determination of the quality of the image.

19. The device of claim 13, wherein the processor determines the warped distance (s′

) according to the equation;

s′

=s−

kAs(s−

1)

where s denotes the distance of the scaled pixel to one of the neighboring pixels, A denotes the symmetry of the scaled pixel, and k denotes the intensity of the filter response for the scaled pixel.

20. The device of claim 13, wherein the processor determines the warped distance (s′

) for the green (G) channel of the image encoded according to the RGB color format, and applies the filter to the red (R), green (G), and blue (B) channels based on the warped distance (s′

) determined for the green (G) channel of the image to perform spatial variant interpolation to determine the value of the scaled pixel.

21. The device of claim 13, wherein the processor determines the warped distance (s′

) for the luminance (Y) channel of the image encoded according to the YCbCr color format, and applies the filter based on the warped distance (s′

) determined for the luminance (Y) channel to perform spatial variant interpolation to determine the value of the scaled pixel.

22. The device of claim 13, wherein the processor applies the filter based on the determined warped distance (s′

) to perform spatial variant bicubic interpolation to determine the value of the scaled pixel.

23. The device of claim 22, wherein the processor applies the filter described by the equation:

24. The device of claim 13, further comprising a display to display upscaled image, wherein the processor generates the upscaled image that includes the value of the scaled pixel.

25. A non-transitory computer-readable medium comprising instructions for causing a programmable processor to:

determine a symmetry (A) of a scaled pixel in an image based on content characteristics of neighboring pixels in the image;

determine an intensity (k) of a spatial variant interpolation filter based on the symmetry (A);

adjust the spatial variant interpolation filter based on the intensity (k), wherein the instructions that cause the processor to adjust the spatial variant interpolation filter based on the intensity (k) comprise instructions that cause the processor to determine a warped distance (s′

) based on the symmetry (A) of the scaled pixel and the intensity (k) of the filter response, and adjust the spatial variant interpolation filter based on the warped distance (s′

); and

apply the spatial variant interpolation filter to interpolate the scaled pixel,

wherein the instructions cause the processor to determine whether the symmetry (A) resides within one of a plurality of zones (A₀-A_n), and set the intensity (k) to one of a plurality of intensity values (k₀-k_n) associated with the one of the plurality of zones (A₀-A_n) in which the symmetry (A) lies.

26. The non-transitory computer-readable medium of claim 25, wherein the instructions cause the processor to determine the symmetry (A) of the scaled pixel according to the equation:

A=(|f(x_n+1)−

f(x<sub>n−

1</sub>)|−

|f(x_n+2)−

f(x_n)|)>

>

8

where the neighboring pixels comprise pixels denoted by x_n+1, x<sub>n−

1</sub>, x_n+2, and x_n, and f(x_n+1), f(x<sub>n−

1</sub>), f(x_n+2), and f(x_n) denote the value associated with the respective pixels x_n+1, x<sub>n−

1</sub>, x_n+2, and x_n.

27. The non-transitory computer-readable medium of claim 25, wherein the instructions cause the processor to selectively set the intensity (k) to equal zero such that the warped distance (s′

) equals a distance (s) from the scaled pixel to one of the neighboring pixels to transform the spatial invariant interpolation filter into a spatial invariant interpolation filter, and apply the spatial invariant interpolation filter to interpolate the scaled pixel.

28. The non-transitory computer-readable medium of claim 25, wherein the instructions cause the processor to access one of a plurality of data structures that store the association of zones (A₀-A_n) to the plurality of intensity values (k₀-k_n) based on the determined symmetry (A) to retrieve one of the plurality of intensity values (k₀-k_n), and set the intensity (k) to the retrieved one of the plurality of intensity values (k₀-k_n).

29. The non-transitory computer-readable medium of claim 28, wherein the instructions cause the processor to access one of the plurality of data structures based on one or more of available system resources, and an application requesting the image.

30. The non-transitory computer-readable medium of claim 25, wherein the instructions cause the processor to determine the quality of the image, and determine the intensity (k) of the filter response for the scaled pixel based on the determination of the quality of the image.

31. The non-transitory computer-readable medium of claim 25, wherein the instructions cause the processor to determine the warped distance (s′

) according to the equation;

s′

=s−

kAs(s−

1)

where s denotes the distance of the scaled pixel to one of the neighboring pixels, A denotes the symmetry of the scaled pixel, and k denotes the intensity of the filter response for the scaled pixel.

32. The non-transitory computer-readable medium of claim 25, wherein the instructions cause the processor to determine the warped distance (s′

) for the green (G) channel of the image encoded according to the RGB color format, and apply the filter to the red (R), green (G), and blue (B) channels based on the warped distance (s′

) determined for the green (G) channel of the image to perform spatial variant interpolation to determine the value of the scaled pixel.

33. The non-transitory computer-readable medium of claim 25, wherein the instructions cause the processor to determine the warped distance (s′

) for the luminance (Y) channel of the image encoded according to the YCbCr color format, and apply the filter to the luminance (Y) channel based on the warped distance (s′

) determined for the luminance (Y) channel to perform spatial variant interpolation to determine the value of the scaled pixel.

34. The non-transitory computer-readable medium of claim 25, wherein the instructions cause the processor to apply the filter based on the determined warped distance (s′

) to perform spatial variant bicubic interpolation to determine the value of the scaled pixel.

35. The non-transitory computer-readable medium of claim 25, wherein the instructions cause the processor to apply the filter described by the equation:

36. The non-transitory computer-readable medium of claim 25, further comprising instructions that cause the processor to generate upscaled image that includes the value of the scaled pixel, drive a display device to display the upscaled image.

37. A device comprising:

means for determining a symmetry (A) of a scaled pixel in an image based on content characteristics of neighboring pixels in the image;

means for determining an intensity (k) of a spatial variant interpolation filter based on the symmetry (A);

means for adjusting the spatial variant interpolation filter based on the intensity (k), wherein the means for adjusting includes means for determining a warped distance (s′

) based on the symmetry (A) of the scaled pixel and the intensity (k) of the filter response, and means for adjusting the spatial variant interpolation filter based on the warped distance (s′

); and

means for applying the spatial variant interpolation filter to interpolate the scaled pixel,

wherein the means for determining the intensity (k) of the filter response comprises;

means for determining whether the symmetry (A) resides within one of a plurality of zones (A₀-A_n); and

means for setting the intensity (k) to one of a plurality of intensity values (k₀-k_n) associated with the one of the plurality of zones (A₀-A_n) in which the symmetry (A) lies.