Video super-resolution by fast video segmentation for boundary accuracy control

• US 9,652,829 B2
• Filed: 09/22/2015
• Issued: 05/16/2017
• Est. Priority Date: 01/22/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Tao, Li, Yang, Bo
• Primary Examiner(s): Ahmed, Samir
• Application Number:
  US14/862,020
• Publication Number:
  US 20160217552A1
• Time in Patent Office: 602 Days
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/54 relating to texture
• CPC Class Codes: G06V 10/56 relating to colour

Abstract

A method includes generating, by a processor, a label map by labeling pixels in an image. A percentage of a first and second type of texture pixels is determined based on the label map to generate a texture map. The texture map is updated. A directional edge map is generated based on a data correlation analysis based on the texture map and the label map. A gain map with boundaries and smoothing is generated using localized pixels based on information from one or more of the label map, the texture map, and the directional edge map. An enhancement controlled upscaled image frame is generated using one or more of the label map, the texture map, the directional edge map and the gain map as a weight factor.

Claims

1. A method comprising:

generating, by a processor, a label map by labeling pixels in an image frame based on a closest color cluster center to the pixels;

determining a percentage of a first type of texture pixels and a percentage of a second type of texture pixels based on the label map to generate a texture map of the image frame;

updating the texture map by updating texture probability as a local average with an updating factor for a particular difference between the percentage of the first type of texture pixels and the percentage of the second type of texture pixels;

generating a directional edge map based on a data correlation analysis by removing a correlation response in texture regions based on the texture map and the label map;

generating a gain map with boundaries and smoothing using localized pixels based on information from one or more of the label map, the texture map, and the directional edge map; and

generating an enhancement controlled upscaled image frame based on using one or more of the label map, the texture map, the directional edge map and the gain map as a weight factor.

2. The method of claim 1, wherein generating the label map further comprises:

generating a color probability map based on a likelihood of pixels of the image frame being a color pixel or a gray pixel; and

separating the image frame into a color portion and a gray portion using the color probability map and a predetermined threshold.

3. The method of claim 2, wherein generating the label map further comprises:

generating histograms of color clusters and gray clusters based on the color portion and the gray portion; and

adaptively selecting peaks from the histograms as color cluster centers.

4. The method of claim 3, wherein color pixels are clustered into the color clusters based on hue channel and gray pixels are clustered into the gray clusters based on value channel.

5. The method of claim 4, wherein the pixels in the label map bear a segmental label of a cluster it belongs to, and the segmental label indicating intensity and distinguishing a color pixel from a gray pixel.

6. The method of claim 5, wherein adaptively selecting peaks from the histograms limits a number of labels from the label map to increase hardware processing speed.

7. The method of claim 1, wherein the first type of texture pixels is based on a first threshold, the second type of texture pixels is based on a second threshold, and the first threshold is greater than the second threshold.

8. The method of claim 6, wherein generating the enhancement controlled upscaled image using the gain map to adjust detail enhancement strengths optimizes high frequency details and reduces ringing artifacts.

9. The method of claim 1, wherein the processor uses the gain map to control enhancement strength of pixel details in super-resolution by distinguishing boundaries between a non-texture pixel region and a texture pixel region.

10. The method of claim 1, wherein the processor uses the directional edge map to limit dilation within pixels with a same label to avoid crossing boundaries between non-texture pixel regions and texture pixel regions, and data correlation strength that indicates probability of a texture pixel region or a non-texture pixel region is bounded to remove errors in texture pixel regions.

11. An apparatus comprising:

a memory device configured to receive an image frame; and

a processor configured to;

generate a label map by labeling pixels in the image frame based on a closest color cluster center to the pixels;

determine a percentage of a first type of texture pixels and a percentage of a second type of texture pixels based on the label map to generate a texture map of the image frame;

update the texture map by updating texture probability as a local average with an updating factor for a particular difference between the percentage of the first type of texture pixels and the percentage of the second type of texture pixels;

generate a directional edge map based on a data correlation analysis by removing a correlation response in texture regions based on the texture map and the label map;

generate a gain map with boundaries and smoothing using localized pixels based on information from one or more of the label map, the texture map, and the directional edge map; and

construct an enhancement controlled upscaled image frame based on using one or more of the label map, the texture map, the directional edge map and the gain map as a weight factor.

12. The apparatus of claim 11, wherein the processor is further configured to:

generate a color probability map based on a likelihood of pixels of the image frame being a color pixel or a gray pixel; and

separate the image frame into a color portion and a gray portion using the color probability map and a predetermined threshold.

13. The apparatus of claim 12, wherein the processor is further configured to:

generate histograms of color clusters and gray clusters based on the color portion and the gray portion; and

adaptively select peaks from the histograms as color cluster centers.

14. The apparatus of claim 13, wherein the processor is configured to cluster color pixels into the color clusters based on hue channel, and to cluster gray pixels into the gray clusters based on value channel.

15. The apparatus of claim 11, wherein the pixels in the label map bear a segmental label of a cluster it belongs to, and the segmental label indicating intensity and distinguishes a color pixel from a gray pixel.

16. The apparatus of claim 15, wherein the processor is configured to adaptively select peaks from the histograms to limit a number of labels from the label map to increase hardware processing speed.

17. The apparatus of claim 11, wherein the first type of texture pixels is based on a first threshold, the second type of texture pixels is based on a second threshold, and the first threshold is greater than the second threshold.

18. The apparatus of claim 17, wherein the processor is configured to generate the enhancement controlled upscaled image using the gain map to adjust detail enhancement strengths to optimize high frequency details and to reduce ringing artifacts.

19. The apparatus of claim 11, wherein the processor is configured to:

use the gain map to control enhancement strength of pixel details in super-resolution to distinguish boundaries between a non-texture pixel region and a texture pixel region; and

use the directional edge map to limit dilation within pixels with a same label to avoid crossing boundaries between non-texture pixel regions and texture pixel regions; and

bound data correlation strength that indicates probability of a texture pixel region or a non-texture pixel region to remove errors in texture pixel regions.

20. The apparatus of claim 11, wherein the processor is embedded in a microchip of an electronic television device.

21. A non-transitory processor-readable medium that includes a program that when executed by a processor performs a method comprising:

generating, by the processor, a label map by labeling pixels in an image frame based on a closest color cluster center to the pixels;

determining a percentage of a first type of texture pixels and a percentage of a second type of texture pixels based on the label map to generate a texture map of the image frame;

updating the texture map by updating texture probability as a local average with an updating factor for a particular difference between the percentage of the first type of texture pixels and the percentage of the second type of texture pixels;

generating a directional edge map based on a data correlation analysis by removing a correlation response in texture regions based on the texture map and the label map;

generating a gain map with boundaries and smoothing using localized pixels based on information from one or more of the label map, the texture map, and the directional edge map; and

generating an enhancement controlled upscaled image frame based on using one or more of the label map, the texture map, the directional edge map and the gain map as a weight factor.

22. The non-transitory processor-readable medium of claim 21, wherein generating the label map further comprises:

generating a color probability map based on a likelihood of pixels of the image frame being a color pixel or a gray pixel; and

separating the image frame into a color portion and a gray portion using the color probability map and a predetermined threshold.

23. The non-transitory processor-readable medium of claim 22, wherein generating the label map further comprises:

generating histograms of color clusters and gray clusters based on the color portion and the gray portion; and

adaptively selecting peaks from the histograms as color cluster centers.

24. The non-transitory processor-readable medium of claim 23, wherein color pixels are clustered into the color clusters based on hue channel and gray pixels are clustered into the gray clusters based on value channel.

25. The non-transitory processor-readable medium of claim 21, wherein the pixels in the label map bear a segmental label of a cluster it belongs to, and the segmental label indicating intensity and distinguishing a color pixel from a gray pixel.

26. The non-transitory processor-readable medium of claim 25, wherein adaptively selecting peaks from the histograms limits a number of labels from the label map to increase hardware processing speed.

27. The non-transitory processor-readable medium of claim 21, wherein the first type of texture pixels is based on a first threshold, the second type of texture pixels is based on a second threshold, and the first threshold is greater than the second threshold.

28. The non-transitory processor-readable medium of claim 26, wherein generating the enhancement controlled upscaled image using the gain map to adjust detail enhancement strengths optimizes high frequency details and reduces ringing artifacts.

29. The non-transitory processor-readable medium of claim 21, wherein the processor uses the gain map to control enhancement strength of pixel details in super-resolution by distinguishing boundaries between a non-texture pixel region and a texture pixel region.

30. The non-transitory processor-readable medium of claim 21, wherein the processor uses the directional edge map to limit dilation within pixels with a same label to avoid crossing boundaries between non-texture pixel regions and texture pixel regions, and data correlation strength that indicates probability of a texture pixel region or a non-texture pixel region is bounded to remove errors in texture pixel regions.