Method and device for performing super-resolution on an input image

• US 9,600,860 B2
• Filed: 04/25/2014
• Issued: 03/21/2017
• Est. Priority Date: 04/25/2013
• Status: Expired due to Fees

• Current Assignee: Magnolia Licensing LLC (IP Edge LLC)
• Original Assignee: Thomson Licensing (Vantiva SA)
• Inventors: Perez Pellitero, Eduardo, Salvador Marcos, Jordi, Kochale, Axel
• Primary Examiner(s): Li, Ruiping
• Application Number:
  US14/786,981
• Publication Number:
  US 20160078600A1
• Time in Patent Office: 1,061 Days
• Field of Search: 382/115
• US Class Current: 1/1
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes

Abstract

A method for performing super-resolution on an input image having low resolution, comprises generating a generic training data set of descriptors extracted from regions of training images, and for each patch of the input image, determining a defined number of nearest neighbor regions, extracting example patches from the nearest neighbor regions and collecting the example patches in an example patch data base, determining a combination of low-resolution example patches that, according to their descriptors, optimally approximate the current patch, and constructing a high-resolution patch, wherein a super-resolved image is obtained.

Claims

1. A method for performing super-resolution on an input image having low resolution, comprising steps of

generating a training data set of descriptors, or retrieving from a storage a previously generated training data set of descriptors, the descriptors being extracted from regions of training images, the training images comprising low-resolution and corresponding high-resolution images, wherein each descriptor comprises a region identifier and a geometrical feature;

dividing the input image into a plurality of input image patches, wherein the patches are smaller than the regions;

for each input image patch, performing the steps of

determining a defined number of nearest neighbor regions, the nearest neighbor regions being low-resolution regions from the training data set that have geometrical features that are most similar to a current input image patch;

from each nearest neighbor region, extracting a plurality of example patches by dense sampling, wherein the dense sampling comprises sampling in regular intervals of r pixels and is independent from the image contents within the region, r being an integer, and collecting the example patches in an example patch data base that is specific for the current input image;

determining from the example patch data base a low-resolution example patch, or a combination of two or more low-resolution example patches, that optimally match geometrical features of the current input image patch; and

constructing a target high-resolution patch from one or more high-resolution patches corresponding to said one or more low-resolution example patches, according to the determined combination,

wherein the target high-resolution patches for all input patches form a super-resolved image.

2. The method according to claim 1, wherein for each input image patch the following steps are performed

a. before said determining a defined number k of nearest neighbor regions, extracting descriptors of a current input image patch;

b. after said determining a defined number k of nearest neighbor regions, extracting a plurality of example patches from said nearest neighbor region; and

c. constructing from said example patches said example patch data base that is specific for the current image.

3. The method according to claim 1, wherein in the step of determining a combination of one or more low-resolution example patches from the example patch data base, the combination is a linear combination of at least two low-resolution example patches.

4. The method according to claim 1, wherein the input image patches are overlapping, and the input image patches are rectangular or square.

5. The method according to claim 1, further comprising a step of optimizing the example patch data base after all example patches are in the example patch data base and before said determining a combination of one or more low-resolution example patches.

6. The method according to claim 1, wherein the patches of the input image have a size between 3×

3 and 5×

5 pixels.

7. The method according to claim 1, wherein said defined number of nearest neighbor patches is k=1.

8. The method according to claims 1, wherein said defined number of nearest neighbor patches is 1<

k<

5.

9. A device for performing super-resolution on an input image having low resolution, comprising

a. a training data memory for storing training images and a descriptor data base for storing descriptors;

b. an input image memory and an output image memory;

c. a descriptor extraction unit for generating a training data set of descriptors extracted from patches of training images, the training images being not specific to the input image , wherein each descriptor comprises a region identifier and a geometrical feature;

d. a k-Nearest-Neighbor (KNN) determining unit for determining a defined number k of nearest neighbor regions, the nearest neighbor regions being low-resolution regions of the training data set that are most similar to a current patch according to the descriptors, wherein the patches are smaller than the regions;

e. an example patch extraction unit for extracting a plurality of example patches from said region by dense sampling, and collecting the example patches in an example patch data base that is specific for the current input image, wherein the dense sampling comprises sampling in regular intervals of r pixels and is independent from the image contents within the region, r being an integer;

f. a determining unit for determining, when all found example patches are in the example patch data base, a combination of one or more low-resolution example patches from the example patch data base that, according to their descriptors, optimally approximate the current patch; and

g. a patch combiner unit for constructing a high-resolution patch of a super-resolved image from one or more high-resolution patches corresponding to said one or more low-resolution example patches, according to the determined combination, wherein a super-resolved image is obtained;

h. a control unit for applying to each patch of the input image the KNN determining unit, example patch extraction unit, determining unit and patch combiner unit;

i. an image output unit for providing the super-resolved image.

10. The device according to claim 9, wherein the example patch extraction unit comprises

a. a descriptor extraction unit for extracting, before said determining a defined number k of nearest neighbor patches, descriptors of a current patch;

b. a patch extraction unit for extracting a plurality of example patches from said region; and

c. a data base construction unit for constructing from said example patches said example patch data base that is specific for the current patch.

11. The device according to claim 9, wherein the combination is a linear combination of at least two low-resolution example patches.

12. The device according to claim 9, further comprising an optimizer unit for optimizing the example patch data base.

13. The device according to claim 9, wherein the patches of the input image have a size between 3×

3 and 5×

5 pixels.

14. A device for performing super-resolution on an input image having low resolution, comprising one or more processors and one or more memories, wherein at least one of the one or more memories has stored thereon instructions that when executed on at least one of the one or more processors configure the one or more processors and the one or more memories to comprise

a. a training data memory for storing training images and a descriptor data base for storing descriptors;

b. an input image memory and an output image memory;

c. a descriptor extraction unit for generating a training data set of descriptors extracted from patches of training images, the training images being not specific to the input image , wherein each descriptor comprises a region identifier and a geometrical feature;

d. a k-Nearest-Neighbor (KNN) determining unit for determining a defined number k of nearest neighbor regions, the nearest neighbor regions being low-resolution regions of the training data set that are most similar to a current patch according to the descriptors, wherein the patches are smaller than the regions;

e. an example patch extraction unit for extracting a plurality of example patches from said region by dense sampling, and collecting the example patches in an example patch data base that is specific for the current input image, wherein the dense sampling comprises sampling in regular intervals of r pixels and is independent from the image contents within the region, r being an integer;

f. a determining unit for determining, when all found example patches are in the example patch data base, a combination of one or more low-resolution example patches from the example patch data base that, according to their descriptors, optimally approximate the current patch; and

g. a patch combiner unit for constructing a high-resolution patch of a super-resolved image from one or more high-resolution patches corresponding to said one or more low-resolution example patches, according to the determined combination, wherein a super-resolved image is obtained;

h. a control unit for applying to each patch of the input image the KNN determining unit, example patch extraction unit, determining unit and patch combiner unit;

i. an image output unit for providing the super-resolved image.

15. The device according to claim 14, wherein the example patch extraction unit comprises

a. a descriptor extraction unit for extracting, before said determining a defined number k of nearest neighbor patches, descriptors of a current patch;

b. a patch extraction unit for extracting a plurality of example patches from said region; and

c. a data base construction unit for constructing from said example patches said example patch data base that is specific for the current patch.

16. The device according to claim 14, wherein the combination is a linear combination of at least two low-resolution example patches.

17. The device according to claim 14, further comprising an optimizer unit for optimizing the example patch data base.

18. The device according to claim 14, wherein the patches of the input image have a size between 3×

3 and 5×

5 pixels.