Adaptive spatial variant interpolation for image upscaling

• US 20080036792A1
• Filed: 08/09/2006
• Published: 02/14/2008
• Est. Priority Date: 08/09/2006
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: El-Maleh, Khaled Helmi, Liang, Yi, Lai, King-Chung
• Granted Patent: US 9,013,511 B2
• Time in Patent Office: Days
• US Class Current: 345/660
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403

Abstract

This disclosure describes adaptive spatial variant interpolation (SVI) techniques for image upscaling. In various embodiments, the interpolation techniques described in this disclosure may support low complexity upscaling of image while promoting high image quality, including enhanced sharpness, higher contrast and more accurate interpolation. The interpolation techniques may be applied using generalized finite impulse response (FIR) filters. In some embodiments, the interpolation techniques may be content-adaptive to provide more accurate interpolation while suppressing significant artifacts associated with sharp edges. In addition, the interpolation techniques may be readily applicable to upscaling of color imagery and video, e.g., in both YCbCr (luminance, blue chrominance, red chrominance) and RGB (red, green, blue) formats.

Claims

1. A method comprising:

determining a symmetry (A) of a scaled pixel in an image based on content characteristics of neighboring pixels in the image;

determining an intensity (k) of a spatial variant interpolation filter based on the symmetry A;

adjusting the spatial variant interpolation filter based on the intensity (k); and

applying the spatial variant interpolation filter to interpolate the scaled pixel.

2. The method of claim 1, wherein adjusting the spatial variant interpolation filter includes determining a warped distance (s′

) based on the symmetry (A) of the scaled pixel and the intensity (k) of the filter response, and adjusting the spatial variant interpolation filter based on the warped distance (s′

).

3. The method of claim 2, wherein determining the symmetry (A) of the scaled pixel comprises determining the symmetry (A) of the scaled pixel according to the equation:

A=(|f(x_n+1)−

f(x<sub>n−

1</sub>)|−

|f(x_n+2)−

f(x_n)|)>

>

8

where the neighboring pixels comprise pixels denoted by x_n+1, x<sub>n−

1</sub>, x_n+2, and x_n, and f(x_n+1), f(x<sub>n−

1</sub>), f(x_n+2), and f(x_n) denote the value associated with the respective pixels x_n+1, x<sub>n−

1</sub>, x_n+2, and x_n.

4. The method of claim 2, wherein determining the intensity (k) of the filter response comprises:

determining whether the symmetry (A) resides within one of a plurality of zones (A₀-A_n);

setting the intensity (k) to one of a plurality of intensity values (k₀-k_n) associated with the one of the plurality of zones (A₀-A_n) in which the symmetry (A) lies.

5. The method of claim 4, further comprising selectively setting the intensity (k) to equal zero such that the warped distance (s′

) equals a distance (s) from the scaled pixel to one of the neighboring pixels to transform the spatial variant interpolation filter into a spatial invariant interpolation filter, and applying the spatial invariant interpolation filter to interpolate the scaled pixel.

6. The method of claim 4, wherein setting the intensity (k) comprises:

accessing one of a plurality of data structures that store the association of zones (A₀-A_n) to the plurality of intensity values (k₀-k_n) based on the determined symmetry (A) to retrieve one of the plurality of intensity values (k₀-k_n); and

setting the intensity (k) to the retrieved one of the plurality of intensity values (k₀-k_n).

7. The method of claim 6, wherein accessing one of the plurality of data structures comprises accessing one of the plurality of data structures based on one or more of available system resources, and a type of an application requesting the image.

8. The method of claim 2, wherein determining the intensity (k) of the filter response comprises:

determining a quality of the image; and

determining the intensity (k) of the filter response for the scaled pixel based on the determination of the quality of the image.

9. The method of claim 2, wherein determining the warped distance (s′

) comprises determining the warped distance (s′

) according to the equation;

s′

=s−

kAs(s−

1)

where s denotes the distance of the scaled pixel to one of the neighboring pixels, A denotes the symmetry of the scaled pixel, and k denotes the intensity of the filter response for the scaled pixel.

10. The method of claim 2,

wherein determining the warped distance (s′

) comprises determining the warped distance (s′

) for the green (G) channel of the image encoded according to the RGB color format, and

wherein applying the filter comprises applying the filter to the red (R), green (G), and blue (B) channels based on the warped distance (s′

) determined for the green (G) channel of the image to perform spatial variant interpolation to determine the value of the scaled pixel.

11. The method of claim 2,

wherein determining the warped distance (s′

) comprises determining the warped distance (s′

) for the luminance (Y) channel of the image encoded according to the YCbCr color format;

wherein applying the filter comprises applying the filter to the luminance (Y) channel based on the warped distance (s′

) determined for the luminance (Y) channel to perform spatial variant interpolation to determine the value of the scaled pixel.

12. The method of claim 2, wherein applying the filter comprises applying a filter based on the determined warped distance (s′

) to perform spatial variant bicubic interpolation to determine the value of the scaled pixel.

13. The method of claim 12, wherein applying the filter further comprises applying the filter described by the equation:

$f<br><br>\left(x\right)=\frac{1}{2}<br><br>\left(-s^{\mathrm{\prime <br><br>3}}+2<br><br>s^{\mathrm{\prime <br><br>2}}-s^{\mathrm{\prime <br><br>}}\right)<br><br>f<br><br>\left(x_{n-1}\right)+\frac{1}{2}<br><br>\left(3<br><br>s^{\mathrm{\prime <br><br>3}}-5<br><br>s^{\mathrm{\prime <br><br>2}}+2\right)<br><br>f<br><br>\left(x_n\right)+\frac{1}{2}<br><br>\left(-3<br><br>s^{\mathrm{\prime <br><br>3}}+4<br><br>s^{\mathrm{\prime <br><br>2}}+s^{\mathrm{\prime <br><br>}}\right)<br><br>f<br><br>\left(x_{n+1}\right)+\frac{1}{2}<br><br>\left(s^{\mathrm{\prime <br><br>3}}-s^{\mathrm{\prime <br><br>2}}\right)<br><br>f<br><br>\left(x_{n+2}\right)$

where f(x) denotes the value of the scaled pixel, s′

denotes the warped distance, the neighboring pixels comprise pixels denoted by x_n+1, x<sub>n−

1</sub>, x_n+2, and x_n, and f(x₊₁), f(x<sub>n−

1</sub>), f(x_n+2), and f(x_n) denote the value associated with the respective pixels x_n+1, x<sub>n−

1</sub>, x_n+2, and x_n.

14. The method of claim 1, further comprising:

generating upscaled image that includes the value of the scaled pixel; and

displaying the upscaled image via a display.

16. The device of claim 1, wherein the processor determines a warped distance (s′

) based on the symmetry (A) of the scaled pixel and the intensity (k) of the filter response, and adjusts the spatial variant interpolation filter based on the warped distance (s′

).

17. The device of claim 16, wherein the processor determines the symmetry (A) of the scaled pixel according to the equation:

A=(|f(x_n+1)−

f(x<sub>n−

1</sub>)|−

|f(x_n+2)−

f(x_n)|)>

>

8

where the neighboring pixels comprise pixels denoted by x_n+1, x<sub>n−

1</sub>, x_n+2, and x_n, and f(x_n+1), f(x<sub>n−

1</sub>), f(x_n+2), and f(x_n) denote the value associated with the respective pixels x_n+1, x<sub>n−

1</sub>, x_n+2, and x_n.

18. The device of claim 16, wherein the processor determines whether the symmetry (A) resides within one of a plurality of zones (A₀-A_n), and sets the intensity (k) to one of a plurality of intensity values (k₀-k_n) associated with the one of the plurality of zones (A₀-A_n) in which the symmetry (A) lies.

19. The device of claim 16, wherein the processor selectively sets the intensity (k) to equal zero such that the warped distance (s′

) equals a distance (s) from the scaled pixel to one of the neighboring pixels to transform the spatial invariant interpolation filter into a spatial invariant interpolation filter, and applies the spatial invariant interpolation filter to interpolate the scaled pixel.

20. The device of claim 18, wherein the memory stores a plurality of data structures that store an association between zones of symmetry (A₀-A_n) and a plurality of intensity values (k₀-k_n), and the processor accesses one of the plurality of data structures that stores the association of zones (A₀-A_n) to the plurality of intensity values (k₀-k_n) based on the determined symmetry (A) to retrieve one of the plurality of intensity values (k₀-k_n), and sets the intensity (k) to the retrieved one of the plurality of intensity values (k₀-k_n).

21. The device of claim 20, wherein the processor accesses one of the plurality of data structures based on one or more of available system resources, and a type of application requesting the image.

22. The device of claim 16, wherein the processor determines quality of the image, and determines the intensity (k) of the filter response for the scaled pixel based on the determination of the quality of the image.

23. The device of claim 16, wherein the processor determines the warped distance (s′

) according to the equation;

s′

=s−

kAs(s−

1)

where s denotes the distance of the scaled pixel to one of the neighboring pixels, A denotes the symmetry of the scaled pixel, and k denotes the intensity of the filter response for the scaled pixel.

24. The device of claim 16, wherein the processor determines the warped distance (s′

) for the green (G) channel of the image encoded according to the RGB color format, and applies the filter to the red (R), green (G), and blue (B) channels based on the warped distance (s′

) determined for the green (G) channel of the image to perform spatial variant interpolation to determine the value of the scaled pixel.

25. The device of claim 16, wherein the processor determines the warped distance (s′

) for the luminance (Y) channel of the image encoded according to the YCbCr color format, and applies the filter based on the warped distance (s′

) determined for the luminance (Y) channel to perform spatial variant interpolation to determine the value of the scaled pixel.

26. The device of claim 16, wherein the processor applies the filter based on the determined warped distance (s′

) to perform spatial variant bicubic interpolation to determine the value of the scaled pixel.

27. The device of claim 26, wherein the processor applies the filter described by the equation:

$f<br><br>\left(x\right)=\frac{1}{2}<br><br>\left(-s^{\mathrm{\prime <br><br>3}}+2<br><br>s^{\mathrm{\prime <br><br>2}}-s^{\mathrm{\prime <br><br>}}\right)<br><br>f<br><br>\left(x_{n-1}\right)+\frac{1}{2}<br><br>\left(3<br><br>s^{\mathrm{\prime <br><br>3}}-5<br><br>s^{\mathrm{\prime <br><br>2}}+2\right)<br><br>f<br><br>\left(x_n\right)+\frac{1}{2}<br><br>\left(-3<br><br>s^{\mathrm{\prime <br><br>3}}+4<br><br>s^{\mathrm{\prime <br><br>2}}+s^{\mathrm{\prime <br><br>}}\right)<br><br>f<br><br>\left(x_{n+1}\right)+\frac{1}{2}<br><br>\left(s^{\mathrm{\prime <br><br>3}}-s^{\mathrm{\prime <br><br>2}}\right)<br><br>f<br><br>\left(x_{n+2}\right)$

where f(x) denotes the value of the scaled pixel, s′

denotes the warped distance, the neighboring pixels comprise pixels denoted by x_n+1, x<sub>n−

1</sub>, x_n+2, and x_n, and f(x_n+1), f(x<sub>n−

1</sub>), f(x_n+2), and f(x_n) denote the value associated with the respective pixels x_n+1, x<sub>n−

1</sub>, x_n+2, and x_n.

28. The device of claim 16, further comprising a display to display upscaled image, wherein the processor generates the upscaled image that includes the value of the scaled pixel.

30. The computer-readable medium of claim 27, wherein the instructions cause the processor to determine a warped distance (s′

) based on the symmetry (A) of the scaled pixel and the intensity (k) of the filter response, and adjust the spatial variant interpolation filter based on the warped distance (s′

).

31. The computer-readable medium of claim 30, wherein the instructions cause the processor to determine the symmetry (A) of the scaled pixel according to the equation:

A=(|f(x_n+1)−

f(x<sub>n−

1</sub>)|−

|f(x_n+2)−

f(x_n)|)>

>

8

where the neighboring pixels comprise pixels denoted by x_n+1, x<sub>n−

1</sub>, x_n+2, and x_n, and f(x_n+1), f(x<sub>n−

1</sub>), f(x_n+2), and f(x_n) denote the value associated with the respective pixels x_n+1, x<sub>n−

1</sub>, x_n+2, and x_n.

32. The computer-readable medium of claim 30, wherein the instructions cause the processor to determine whether the symmetry (A) resides within one of a plurality of zones (A₀-A_n), and set the intensity (k) to one of a plurality of intensity values (k₀-k_n) associated with the one of the plurality of zones (A₀-A_n) in which the symmetry (A) lies.

33. The computer-readable medium of claim 30, wherein the instructions cause the processor to selectively set the intensity (k) to equal zero such that the warped distance (s′

) equals a distance (s) from the scaled pixel to one of the neighboring pixels to transform the spatial invariant interpolation filter into a spatial invariant interpolation filter, and apply the spatial invariant interpolation filter to interpolate the scaled pixel.

34. The computer-readable medium of claim 30, wherein the instructions cause the processor to access one of a plurality of data structures that store the association of zones (A₀-A_n) to the plurality of intensity values (k₀-k_n) based on the determined symmetry (A) to retrieve one of the plurality of intensity values (k₀-k_n), and set the intensity (k) to the retrieved one of the plurality of intensity values (k₀-k_n).

35. The computer-readable medium of claim 34, wherein the instructions cause the processor to access one of the plurality of data structures based on one or more of available system resources, and an application requesting the image.

36. The computer-readable medium of claim 30, wherein the instructions cause the processor to determine the quality of the image, and determine the intensity (k) of the filter response for the scaled pixel based on the determination of the quality of the image.

37. The computer-readable medium of claim 30, wherein the instructions cause the processor to determine the warped distance (s′

) according to the equation;

s′

=s−

kAs(s−

1)

where s denotes the distance of the scaled pixel to one of the neighboring pixels, A denotes the symmetry of the scaled pixel, and k denotes the intensity of the filter response for the scaled pixel.

38. The computer-readable medium of claim 30, wherein the instructions cause the processor to determine the warped distance (s′

) for the green (G) channel of the image encoded according to the RGB color format, and apply the filter to the red (R), green (G), and blue (B) channels based on the warped distance (s′

) determined for the green (G) channel of the image to perform spatial variant interpolation to determine the value of the scaled pixel.

39. The computer-readable medium of claim 30, wherein the instructions cause the processor to determine the warped distance (s′

) for the luminance (Y) channel of the image encoded according to the YCbCr color format, and apply the filter to the luminance (Y) channel based on the warped distance (s′

) determined for the luminance (Y) channel to perform spatial variant interpolation to determine the value of the scaled pixel.

40. The computer-readable medium of claim 30, wherein the instructions cause the processor to apply the filter based on the determined warped distance (s′

) to perform spatial variant bicubic interpolation to determine the value of the scaled pixel.

41. The computer-readable medium of claim 30, wherein the instructions cause the processor to apply the filter described by the equation:

$f<br><br>\left(x\right)=\frac{1}{2}<br><br>\left(-s^{\mathrm{\prime <br><br>3}}+2<br><br>s^{\mathrm{\prime <br><br>2}}-s^{\mathrm{\prime <br><br>}}\right)<br><br>f<br><br>\left(x_{n-1}\right)+\frac{1}{2}<br><br>\left(3<br><br>s^{\mathrm{\prime <br><br>3}}-5<br><br>s^{\mathrm{\prime <br><br>2}}+2\right)<br><br>f<br><br>\left(x_n\right)+\frac{1}{2}<br><br>\left(-3<br><br>s^{\mathrm{\prime <br><br>3}}+4<br><br>s^{\mathrm{\prime <br><br>2}}+s^{\mathrm{\prime <br><br>}}\right)<br><br>f<br><br>\left(x_{n+1}\right)+\frac{1}{2}<br><br>\left(s^{\mathrm{\prime <br><br>3}}-s^{\mathrm{\prime <br><br>2}}\right)<br><br>f<br><br>\left(x_{n+2}\right)$

where f(x) denotes the value of the scaled pixel, s′

denotes the warped distance, the neighboring pixels comprise pixels denoted by x_n+1, x<sub>n−

1</sub>, x_n+2, and x_n, and f(x_n+1), f(x<sub>n−

1</sub>), f(x_n+2), and f(x_n) denote the value associated with the respective pixels x_n+1, x<sub>n−

1</sub>, x_n+2, and x_n.

42. The computer-readable medium of claim 30, further comprising instructions that cause the processor to generate upscaled image that includes the value of the scaled pixel, drive a display device to display the upscaled image.

15. A device comprising:

a memory that stores an image; and

a processor that determines a symmetry (A) of a scaled pixel in an image based on content characteristics of neighboring pixels in the image, determines an intensity (k) of an interpolation filter based on the symmetry A, adjusts the interpolation filter based on the intensity (k), and applies the spatial variant interpolation filter to interpolate the scaled pixel.

29. A computer-readable medium comprising instructions for causing a programmable processor to:

determine a symmetry (A) of a scaled pixel in an image based on content characteristics of neighboring pixels in the image;

determine an intensity (k) of a spatial variant interpolation filter based on the symmetry A;

adjust the spatial variant interpolation filter based on the intensity (k); and

apply the spatial variant interpolation filter to interpolate the scaled pixel.

43. A device comprising:

means for determining a symmetry (A) of a scaled pixel in an image based on content characteristics of neighboring pixels in the image;

means for determining an intensity (k) of a spatial variant interpolation filter based on the symmetry A;

means for adjusting the spatial variant interpolation filter based on the intensity (k); and

means for applying the spatial variant interpolation filter to interpolate the scaled pixel.

44. The device of claim 43, wherein the adjusting means includes means for determining a warped distance (s′

) based on the symmetry (A) of the scaled pixel and the intensity (k) of the filter response, and means for adjusting the spatial variant interpolation filter based on the warped distance (s′

).