METHOD AND APPARATUS FOR PERFORMING HIERARCHICAL SUPER-RESOLUTION OF AN INPUT IMAGE

• US 20150093045A1
• Filed: 03/04/2013
• Published: 04/02/2015
• Est. Priority Date: 03/05/2012
• Status: Active Grant

• Current Assignee: InterDigital Madison Patent Holdings (InterDigital, Inc.)
• Original Assignee: Thomson Licensing (Vantiva SA)
• Inventors: Thoreau, Dominique, Guillotel, Philippe, Turkan, Mehmet
• Granted Patent: US 9,652,830 B2
• Time in Patent Office: Days
• US Class Current: 382/299
• CPC Class Codes: G06T 2207/20052 Discrete cosine transform [DCT]
• CPC Class Codes: G06T 2207/20216 Image averaging
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 3/4076 using the original low-resolution images to iteratively correct the high-resolution images
• CPC Class Codes: G06T 3/4084 in the transform domain, e.g. fast Fourier transform [FFT] domain scaling
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction

Abstract

Super-resolution refers to a process of recovering the missing high-frequency details of a given low-resolution image. Known single image SR algorithms are often computationally intractable or unusable for most of the practical applications. The invention relates to a method for performing hierarchical super-resolution based on self content neighboring patches information is based on pyramidal decomposition. The intrinsic geometric property of an input LR patch neighborhood is obtained from the input LR patch and its K nearest neighbors taken from different down-scaled versions of the LR image.

Claims

1-15. -15. (canceled)

16. A method for performing hierarchical super-resolution of an input image, comprising steps of

dividing the input image into patches;

performing spatial decomposition of the input image to at least two lower decomposition levels, wherein at least two lower decomposition level images are obtained;

generating an empty upsampled frame, wherein for each patch of the input image a corresponding upsampled patch in the upsampled frame is generated; and

for each current patch of the input image, performing the steps of

searching in the lower decomposition level images one or more similar patches of same size as the current patch;

for each of the similar patches found in the searching step, determining its respective parent patch in the next higher decomposition level, wherein the parent patches are larger than the current patch;

weighting the determined parent patches, wherein a weight used for weighting a patch is determined from a sparsity of the patch, wherein the sparsity corresponds to a number of non-zero DCT coefficients in the patch, and wherein weighted determined parent patches are obtained;

accumulating the weighted determined parent patches, wherein an upsampled high-resolution patch is obtained; and

replacing, in the up-sampled frame, an upsampled patch corresponding to the current patch with the upsampled high-resolution patch.

17. Method according to claim 16, wherein, in the step of searching in the lower decomposition level images one or more similar patches, a mean of each current patch and a mean of each similar patch is calculated and subtracted from each pixel value of the respective patch, and wherein in the step of replacing, the mean of each current patch is added to each pixel value of the upsampled patch corresponding to the current patch.

18. Method according to claim 16, wherein, in the step of searching in the lower decomposition level images one or more similar patches, a similarity is determined according to a luminance of the pixels in the patches.

19. Method according to claim 16, wherein, in the step of searching in the lower decomposition level images one or more similar patches, a similarity is determined according to a luminance gradient of the patches.

20. Method according to claim 16, wherein the patches the input image are partially overlapping and the corresponding upsampled patches in the upsampled image are partially overlapping.

21. Method according to claim 16, wherein the steps of weighting and accumulating comprise calculating a weighted combination, wherein weights for said weighted combination are determined by solving a constrained least square problem.

22. Method according to claim 16, wherein the pyramidal super-resolution algorithm is a neighbors embedding algorithm.

23. Method according to claim 16, further comprising an additional step of performing back-projection.

24. Method according to claim 16, wherein the parent patch in the next higher decomposition level is determined according to its relative coordinates.

29. Computer-readable storage medium comprising program data that when executed on a processor cause the processor to perform a method according to claim 16.

25. A method for averaging of pixels in performing super-resolution by using block-based predictions, wherein Local Linear Embedding is used and wherein the pixels are from source blocks that are overlapping, the method comprising steps of

determining sparsity factors of the source blocks;

combining the pixels from the source blocks according to weighting factors, wherein the sparsity factor of each source block is used as a weighting factor for its pixels.

26. An apparatus for performing hierarchical super-resolution of an input image, wherein the input image is divided into patches, the apparatus comprising

spatial decomposition unit for performing spatial decomposition of the input image to at least two lower decomposition levels, wherein at least two lower decomposition level images are obtained;

upsampling unit for generating an empty upsampled frame, wherein for each patch of the input image a corresponding upsampled patch in the upsampled frame is generated; and

processing unit for performing for each current patch of the input image the steps of

in a search unit, searching in the lower decomposition level images one or more similar patches of same size as the current patch;

for each of the similar patches found in the searching step, determining in a parent patch determining unit its respective parent patch in the next higher decomposition level, wherein the parent patches are larger than the current patch;

in a weighting unit, weighting the determined parent patches, wherein a weight used for weighting a patch is determined from a sparsity of the patch, wherein the sparsity corresponds to a number of non-zero DCT coefficients in the patch, and wherein weighted determined parent patches are obtained;

in an accumulation unit, accumulating the weighted determined parent patches, wherein an upsampled high-resolution patch is obtained; and

replacing in an insertion unit, in the up-sampled frame, an upsampled patch corresponding to the current patch with the upsampled high-resolution patch.

27. The apparatus according to claim 26, wherein, in the search unit for searching in the lower decomposition level images one or more similar patches, a mean of each current patch and a mean of each similar patch is calculated and subtracted from each pixel value of the respective patch, and wherein in the insertion unit the mean of each current patch is added to each pixel value of the upsampled patch corresponding to the current patch.

28. An apparatus for averaging of pixels in performing super-resolution by using block-based predictions, wherein Local Linear Embedding is used and wherein the pixels are from source blocks that are overlapping, the apparatus comprising

a first processing unit for determining sparsity factors of the source blocks;

and

a second processing unit for combining the pixels from the source blocks according to weighting factors, wherein the sparsity factor of each source block is used as a weighting factor for its pixels.