Method and apparatus for performing hierarchical super-resolution of an input image

• US 9,652,830 B2
• Filed: 03/04/2013
• Issued: 05/16/2017
• Est. Priority Date: 03/05/2012
• Status: Active Grant

• Current Assignee: InterDigital Madison Patent Holdings (InterDigital, Inc.)
• Original Assignee: Thomson Licensing Dtv
• Inventors: Turkan, Mehmet, Guillotel, Philippe, Thoreau, Dominique
• Primary Examiner(s): Dang, Duy M
• Application Number:
  US14/382,569
• Publication Number:
  US 20150093045A1
• Time in Patent Office: 1,534 Days
• CPC Class Codes: G06T 2207/20052 Discrete cosine transform [DCT]
• CPC Class Codes: G06T 2207/20216 Image averaging
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 3/4076 using the original low-resolution images to iteratively correct the high-resolution images
• CPC Class Codes: G06T 3/4084 in the transform domain, e.g. fast Fourier transform [FFT] domain scaling
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction

Abstract

Super-resolution refers to a process of recovering the missing high-frequency details of a given low-resolution image. Known single image SR algorithms are often computationally intractable or unusable for most of the practical applications. The invention relates to a method for performing hierarchical super-resolution based on self content neighboring patches information is based on pyramidal decomposition. The intrinsic geometric property of an input LR patch neighborhood is obtained from the input LR patch and its K nearest neighbors taken from different down-scaled versions of the LR image.

Claims

1. A method for performing hierarchical super-resolution of an input image, comprising:

dividing the input image into patches;

performing spatial decomposition of the input image to at least two lower decomposition levels, wherein at least two lower decomposition level images are obtained; and

for each current patch of the input image,

searching in the lower decomposition level images one or more similar patches of same size as the current patch;

for each of the similar patches found in the searching, determining its respective parent patch in the next higher decomposition level, wherein the parent patches are larger than the current patch;

obtaining weighted determined parent patches using a weight determined from a sparsity of the patch, wherein the sparsity corresponds to a number of non-zero DCT coefficients in the patch;

accumulating the weighted determined parent patches to obtain an upsampled high-resolution patch; and

replacing an upsampled patch of an upsampled frame corresponding to the current patch with the upsampled high-resolution patch.

2. The method according to claim 1, wherein searching in the lower decomposition level images one or more similar patches, further comprised calculating and subtracting a mean of each current patch and a mean of each similar patch from each pixel value of the respective patch, and wherein in the replacing further comprising adding the mean of each current patch to each pixel value of the upsampled patch corresponding to the current patch.

3. The method according to claim 1, wherein the searching in the lower decomposition level images one or more similar patches further comprises determining a similarity according to a luminance of the pixels in the patches.

4. The method according to claim 1, wherein the searching in the lower decomposition level images one or more similar patches further comprises determining a similarity according to a luminance gradient of the patches.

5. The method according to claim 1, wherein the patches the input image are partially overlapping and the corresponding upsampled patches in the upsampled image are partially overlapping.

6. The method according to claim 1, wherein the weighting and accumulating further comprise calculating a weighted combination, wherein weights for said weighted combination are determined by solving a constrained least square problem.

7. The method according to claim 1, wherein the pyramidal super-resolution algorithm is a neighbors embedding algorithm.

8. The method according to claim 1, further comprising performing back-projection.

9. The method according to claim 1, wherein the parent patch in the next higher decomposition level is determined according to its relative coordinates.

10. A non-transitory Computer-readable storage medium comprising program data that when executed on a processor cause the processor to perform the method according to claim 1.

11. An apparatus for performing hierarchical super-resolution of an input image, wherein the input image is divided into patches, the apparatus comprising:

a spatial decomposition unit for performing spatial decomposition of the input image to obtain at least two lower decomposition levels; and

a processing unit adapted to, for each current patch of the input image,

search, in a search unit, in the lower decomposition level images one or more similar patches of same size as the current patch;

for each of the similar patches found in the searching,

determine in a parent patch determining unit its respective parent patch in the next higher decomposition level, wherein the parent patches are larger than the current patch;

weight, in a weighting unit, the determined parent patches, wherein a weight used for weighting a patch is determined from a sparsity of the patch, wherein the sparsity corresponds to a number of non-zero DCT coefficients in the patch, and wherein weighted determined parent patches are obtained;

in an accumulation unit,

accumulate the weighted determined parent patches to obtain an upsampled high-resolution patch; and

replace, in an insertion unit, an upsampled patch of an upsampled frame corresponding to the current patch with the upsampled high-resolution patch.

12. The apparatus according to claim 11, wherein, the processing unit in the search unit is further configured to calculate and subtract a mean of each current patch and a mean of each similar patch from each pixel value of the respective patch, and wherein the processing unit in the insertion unit is further configured to add the mean of each current patch to each pixel value of the upsampled patch corresponding to the current patch.