METHOD AND APPARATUS FOR GENERATING A SUPER-RESOLVED IMAGE FROM A SINGLE IMAGE

• US 20160063677A1
• Filed: 03/26/2014
• Published: 03/03/2016
• Est. Priority Date: 03/27/2013
• Status: Active Grant

• Current Assignee: Magnolia Licensing LLC (IP Edge LLC)
• Original Assignee: Thomson Licensing (Vantiva SA)
• Inventors: KOCHALE, Axel, SALVADOR MARCOS, Jordi, DEAMO, Alberto
• Granted Patent: US 9,760,977 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 3/4092 Image resolution transcoding, e.g. by using client-server architectures
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

Known methods for generating super-resolved images from single input images have various disadvantages. An improved method for generating a super-resolved image from a single low-resolution input image comprises up-scaling the input image to generate an initial version of the super-resolved image, searching, for each patch of the low-resolution input image, similar low-resolution patches in first search windows within down-sampled versions of the input image, and determining, in less down-sampled versions of the input image, high-resolution patches that correspond to the similar low-resolution patches. The determined high-resolution patches are cropped, a second search window is determined in the initial version of the super-resolved image, and a best-matching position for each cropped high-resolution patch is searched within the second search window. Finally, each cropped high-resolution patch is added to the super-resolved image at its respective best-matching position.

Claims

1. A method for generating a super-resolved image from a low-resolution input image, comprising steps of

generating an initial super-resolved image by upsampling the input image;

generating multiple down-scaled versions of the input image;

searching in first search windows within the down-scaled versions of the input image for patches similar to patches of the input image;

searching, for each patch that is found within a first down-scaled version of the input image and that is similar to a patch of the input image, a corresponding upscaled patch within a second down-scaled version of the input image that is larger than the first down-scaled version of the input image;

cropping the upscaled patches, wherein the outermost pixels of at least one edge of the upscaled patches are discarded; and

adding the cropped upscaled patches to the initial super-resolved image, wherein a weighted combination of the cropped upscaled patches to the initial super-resolved image is generated, and wherein the position of the cropped upscaled patches is determined within second search windows that are centred at projected positions of the centres of their corresponding patches of the input image to the scale of the initial super-resolved image.

2. The method of claim 1, wherein

the generated down-scaled versions of the input image are at different scales;

the input image is separated into a plurality of overlapping patches; and

for the patches of the input image,

in the step of searching in first search windows, for a current patch of the input image, determining a corresponding position in the initial super-resolved image;

in the step of searching corresponding upscaled patches, searching, for the current patch of the input image k most similar patches within a first search window in each down-sampled version of the input image, k being a pre-defined number and k being one or more;

for each found patch being one of said k most similar patches in any particular down-sampled version of the input image, determining a patch at a corresponding position in a different version of the input image that has the next higher resolution than the particular down-sampled version; and

defining a second search window within the initial super-resolved image, the second search window being around said determined position corresponding to the current patch;

in the step of cropping the upscaled patches, cropping each determined patch from said different version of the input image, wherein pixels of at least one edge of the patch are removed;

in the step of adding the cropped upscaled patches to the initial super-resolved image, searching for each determined and cropped patch a best-matching position within the second search window, wherein the determined patches are compared with a portion of the initial super-resolved image that is within the second search window; and

adding each determined and cropped patch to the initial super-resolved image at the best-matching position.

3. The method according to claim 1, wherein the second search window has the same size as each determined patch from said different version of the input image, before the cropping.

4. The method according to claim 1, wherein the size of the first search window is 4-6 times the size of each determined from said different version of the input image, before the cropping.

5. The method according to claim 1, wherein in the step of generating an initial super-resolved image by upsampling the input image, the contents of the higher layer are determined by resizing the lower layer by a factor C and then deblurring the resized lower layer.

6. An apparatus for generating a super-resolved image from a low-resolution input image, comprising

an initial upsampling unit for generating an initial super-resolved image by upsampling the input image;

at least one downsampling unit for generating a plurality of down-sampled versions at different scales of the input image;

a patch separation unit for separating the input image into a plurality of overlapping patches;

one or more first search units for searching, for a current patch of the input image, k most similar patches within a first search window in each down-sampled version of the input image, k being a pre-defined number;

an upsampled patch determining unit for determining, for each found patch being one of said k most similar patches in any particular down-sampled version of the input image, a patch at a corresponding position in a less down-sampled version of the input image;

a search window determining unit for defining a second search window within the initial super-resolved image, the second search window being around a position that corresponds to the position of the current patch of the input image;

a patch cropping unit for cropping each determined patch from said less down-sampled version of the input image, wherein pixels of at least one edge of the patch are removed;

one or more second search units for searching, for each determined and cropped patch, a best-matching position within the second search window, wherein the determined patches are compared with a portion of the initial super-resolved image that is within the second search window; and

patch adding unit for adding each determined and cropped patch to the initial super-resolved image at the best-matching position, wherein a weighted combination of the cropped upscaled patches to the initial super-resolved image is generated, and wherein the position of the cropped patches is determined within second search windows that are centred at projected positions of the centres of their corresponding patches of the input image to the scale of the initial super-resolved image.

7. The apparatus of claim 6, further comprising a first search window determining unit for determining a first search window in each down-sampled version of the input image.

8. A computer readable storage medium having executable instructions to cause a computer to perform a method for generating a super-resolved image from a low-resolution input image, comprising

generating an initial super-resolved image by upsampling the input image;

generating multiple down-scaled versions of the input image;

searching in first search windows within the down-scaled versions of the input image for patches similar to patches of the input image;

searching corresponding upscaled patches;

cropping the upscaled patches; and

adding the cropped upscaled patches to the initial super-resolved image, wherein a weighted combination of the cropped upscaled patches to the initial super-resolved image is generated, and wherein the position of the cropped upscaled patches is determined within second search windows that are centred at projected positions of the centres of their corresponding patches of the input image to the scale of the initial super-resolved image.

9. The computer readable storage medium of claim 8, wherein

the generated down-scaled versions of the input image are at different scales;

the input image is separated into a plurality of overlapping patches; and

for the patches of the input image,

in the step of searching in first search windows, for a current patch of the input image, determining a corresponding position in the initial super-resolved image;

in the step of searching corresponding upscaled patches, searching, for the current patch of the input image, k most similar patches within a first search window in each down-sampled version of the input image, k being a pre-defined number and k being one or more;

for each found patch being one of said k most similar patches in any particular down-sampled version of the input image, determining a patch at a corresponding position in a different version of the input image that has the next higher resolution than the particular down-sampled version; and

defining a second search window within the initial super-resolved image, the second search window being around said determined position corresponding to the current patch;

in the step of cropping the upscaled patches, cropping each determined patch from said different version of the input image, wherein pixels of at least one edge of the patch are removed;

in the step of adding the cropped upscaled patches to the initial super-resolved image, searching for each determined and cropped patch a best-matching position within the second search window, wherein the determined patches are compared with a portion of the initial super-resolved image that is within the second search window; and

adding each determined and cropped patch to the initial super-resolved image at the best-matching position.

10. The computer readable storage medium of claim 8, wherein the second search window has the same size as each determined patch from said different version of the input image, before the cropping.

11. The computer readable storage medium of claim 8, wherein the size of the first search window is 4-6 times the size of each determined from said different version of the input image, before the cropping.

12. The computer readable storage medium of claim 8, wherein in generating an initial super-resolved image by upsampling the input image, the contents of the higher layer are determined by resizing the lower layer by a factor C and then deblurring the resized lower layer.