Stable video super-resolution by edge strength optimization

• US 9,478,007 B2
• Filed: 10/19/2015
• Issued: 10/25/2016
• Est. Priority Date: 01/21/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Yang, Bo, Tao, Li
• Primary Examiner(s): MOTSINGER, SEAN T
• Application Number:
  US14/862,005
• Publication Number:
  US 20160210727A1
• Time in Patent Office: 372 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 3/403 Edge-driven scaling; Edge-based scaling
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening

Abstract

A method includes determining, by a hardware processor and based on an image frame, a first edge strength distribution on a horizontal edge map and a second edge strength distribution on a vertical edge map. The first and second edge strength distributions are redistributed to narrow at least one of the first and second edge strength distributions. Non-texture regions of pixels are determined based on a data correlation map of the image frame. Edge strength magnitudes for the pixels of the non-texture regions of the horizontal edge map and the vertical edge map are determined. A high resolution frame is generated by adjusting intensity of respective pixels of the non-texture region of the horizontal edge map and the vertical edge map, the adjusting being based on neighboring pixels edge strength magnitudes.

Claims

1. A method comprising:

determining, by a hardware processor and based on an image frame, a first edge strength distribution on a horizontal edge map and a second edge strength distribution on a vertical edge map;

redistributing the first and second edge strength distributions to narrow at least one of the first and second edge strength distributions;

determining non-texture regions of pixels based on a data correlation map of the image frame;

determining edge strength magnitudes for the pixels of the non-texture regions of the horizontal edge map and the vertical edge map, wherein the edge strength magnitudes for the non-texture regions of the horizontal edge map and the vertical edge map are proportional to the strengths of the data correlation map; and

generating a high resolution frame by adjusting intensity of respective pixels of the non-texture region of the horizontal edge map and the vertical edge map, the adjusting being based on neighboring pixels edge strength magnitudes.

2. The method of claim 1, wherein the image frame is an upscaled image frame.

3. The method of claim 2, wherein redistributing the first and second edge strength distributions further comprises maintaining a sum of the respective first and second edge strength distributions.

4. The method of claim 3, further comprising:

determining non-texture regions of pixels based on an edge data correlation map of pixels of the image frame, wherein the edge data correlation map comprises strength of a correlation of local pixel edge directions.

5. The method of claim 2, further comprising:

extracting the horizontal edge map and the vertical edge map from the image frame.

6. The method of claim 2, wherein for each distribution of the first and second edge strength distributions:

each pixel has a same sign of edge strength;

each pixel has a magnitude of strength larger than a threshold; and

an overall magnitude of each distribution increases monotonically to a peak value and then decreases monotonically.

7. The method of claim 6, wherein for at least two iterations of adjusting intensity of respective pixels, pixel intensities move closer to pixel intensities of the image frame while the edge strengths move closer to edge strengths of the adjusted horizontal and vertical edge maps.

8. The method of claim 1, further comprising:

separating each horizontal and vertical edge map element into an edge pixel sign {1,−

1} and edge magnitude,

wherein the first edge distribution and the second edge distribution each comprise a set of neighboring edge values in a same pixel row or pixel column with a same edge pixel sign and with an edge magnitude that monotonically increases and then monotonically decreases.

9. The method of claim 8, further comprising:

reducing edge magnitude values on two sides of an edge; and

adding the reduced edge magnitude values to center magnitudes.

10. The method of claim 1, wherein edge sharpening is separately applied in a horizontal direction and a vertical direction to reduce computational processing and to reduce required memory size.

11. An apparatus comprising:

a memory device configured to receive an image frame; and

a processor configured to;

extract a first edge strength distribution on a horizontal edge map and a second edge strength distribution on a vertical edge map from the image frame;

redistribute the first and second edge strength distributions to narrow at least one of the first and second edge strength distributions;

determine non-texture regions of pixels based on a data correlation map of the image frame;

determine edge strength magnitudes for the pixels of the non-texture regions of the horizontal edge map and the vertical edge map, wherein the edge strength magnitudes for the non-texture regions of the horizontal edge map and the vertical edge map are proportional to the strengths of the data correlation map; and

generate a high resolution frame by adjusting intensity of respective pixels of the non-texture region of the horizontal edge map and the vertical edge map, the adjusting being based on neighboring pixels edge strength magnitudes.

12. The apparatus of claim 11, wherein the image frame is an upscaled image frame.

13. The apparatus of claim 12, wherein the processor is configured to redistribute the first and second edge strength distributions based on maintaining a sum of the respective first and second edge strength distributions.

14. The apparatus of claim 13, wherein the processor is further configured to determine non-texture regions of pixels based on an edge data correlation map of pixels of the image frame, and the edge data correlation map comprises strength of a correlation of local pixel edge directions.

15. The apparatus of claim 12, wherein for each distribution of the first and second edge strength distributions:

each pixel has a same sign of edge strength;

each pixel has a magnitude of strength larger than a threshold; and

an overall magnitude of each distribution increases monotonically to a peak value and then decreases monotonically.

16. The apparatus of claim 15, wherein the processor is configured to perform at least two iterations to adjust intensity of respective pixels, and for the at least two iterations pixel intensities move closer to pixel intensities of the image frame while the edge strengths move closer to edge strengths of the adjusted horizontal and vertical edge maps.

17. The apparatus of claim 11, wherein the processor is configured to:

separate each horizontal and vertical edge map element into an edge pixel sign {1,−

1} and edge magnitude, and the first edge distribution and the second edge distribution each comprise a set of neighboring edge values in a same pixel row or pixel column with a same edge pixel sign and with an edge magnitude that monotonically increases and then monotonically decreases.

18. The apparatus of claim 17, wherein the processor is configured to:

reduce edge magnitude values on two sides of an edge; and

add the reduced edge magnitude values to center magnitudes.

19. The apparatus of claim 11, wherein the processor is configured to separately apply edge sharpening in a horizontal direction and a vertical direction to reduce computational processing and to reduce memory storage for the memory device.

20. The apparatus of claim 11, wherein the processor is embedded in a microchip of an electronic television device.

21. A non-transitory processor-readable medium that includes a program that when executed by a processor performs a method comprising:

determining, by a hardware processor and based on an image frame, a first edge strength distribution on a horizontal edge map and a second edge strength distribution on a vertical edge map;

redistributing the first and second edge strength distributions to narrow at least one of the first and second edge strength distributions;

determining non-texture regions of pixels based on a data correlation map of the image frame;

determining edge strength magnitudes for the pixels of the non-texture regions of the horizontal edge map and the vertical edge map, wherein the edge strength magnitudes for the non-texture regions of the horizontal edge map and the vertical edge map are proportional to the strengths of the data correlation map; and

generating a high resolution frame by adjusting intensity of respective pixels of the non-texture region of the horizontal edge map and the vertical edge map, the adjusting being based on neighboring pixels edge strength magnitudes.

22. The non-transitory processor-readable medium of claim 21, wherein the image frame is an upscaled image frame.

23. The non-transitory processor-readable medium of claim 22, wherein redistributing the first and second edge strength distributions further comprises maintaining a sum of the respective first and second edge strength distributions.

24. The non-transitory processor-readable medium of claim 23, further comprising:

determining non-texture regions of pixels based on an edge data correlation map of pixels of the image frame, wherein the edge data correlation map comprises strength of a correlation of local pixel edge directions.

25. The non-transitory processor-readable medium of claim 22, further comprising:

extracting the horizontal edge map and the vertical edge map from the image frame.

26. The non-transitory processor-readable medium of claim 22, wherein for each distribution of the first and second edge strength distributions:

each pixel has a same sign of edge strength;

each pixel has a magnitude of strength larger than a threshold; and

an overall magnitude of each distribution increases monotonically to a peak value and then decreases monotonically.

27. The non-transitory processor-readable medium of claim 26, wherein for at least two iterations of adjusting intensity of respective pixels, pixel intensities move closer to pixel intensities of the image frame while the edge strengths move closer to edge strengths of the adjusted horizontal and vertical edge maps.

28. The non-transitory processor-readable medium of claim 21, further comprising:

separating each horizontal and vertical edge map element into an edge pixel sign {1,−

1} and edge magnitude,

wherein the first edge distribution and the second edge distribution each comprise a set of neighboring edge values in a same pixel row or pixel column with a same edge pixel sign and with an edge magnitude that monotonically increases and then monotonically decreases.

29. The non-transitory processor-readable medium of claim 28, further comprising:

reducing edge magnitude values on two sides of an edge; and

adding the reduced edge magnitude values to center magnitudes.

30. The non-transitory processor-readable medium of claim 21, wherein edge sharpening is separately applied in a horizontal direction and a vertical direction to reduce computational processing and to reduce required memory size.