Upscaling Natural Images

• US 20130223734A1
• Filed: 02/24/2012
• Published: 08/29/2013
• Est. Priority Date: 02/24/2012
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Tuzel, Oncel, Porikli, Fatih, Hegde, Chinmay
• Granted Patent: US 8,620,073 B2
• Time in Patent Office: Days
• US Class Current: 382/167
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution

Abstract

A natural input image is upscaled, first by interpolation. Second, edges in the interpolated image are sharpened by a lion-parametric patch transform. The result is decomposed into an edge layer and a detail layer. Only pixels in the detail layer enhanced, and the enhanced detail layer is merged with the edge layer to produce a high resolution version of the input image.

Claims

1. A method for upscaling an input image, comprising the steps of:

interpolating the input image to produce a first intermediate image;

sharpening edges in the first intermediate by replacing patches of pixels with a combination of reference patches to obtain a second intermediate image;

decomposing the second intermediate image into an edge layer and a detail layer;

enhancing the detail layer using a mapping function to produce an enhanced detail layer; and

merging the edge layer and the enhanced detail layer to produce an output image at a larger resolution than the input image, wherein the steps are performed in a processor.

2. The method of claim 1, wherein the reference patches are selected from the input image or the first intermediate image.

3. The method of claim 1, wherein the interpolation uses bicubic interpolation.

4. The method of claim 1, wherein the sharpening uses a patch transform P(I), and the patches are r×

r overlapping pixels.

5. The method of claim 1, wherein a pixel in the input image is replaced by a mean intensity of the pixels in each patch that contains the pixel.

6. The method of claim 4, wherein the patch transform is non-parametric and local to the patches.

7. The method of claim 1, further comprising:

removing a DC component of each patch before performing a nearest neighbor optimization; and

reducing blocking artifacts in each patch.

8. The method of claim 1, wherein variances of intensities of the pixels in each patch that contains the pixel is stored and high variance pixels correspond to the detail layer.

9. The method of claim 1, wherein the decomposing uses weighted least squares.

10. The method of claim 1, wherein the decomposing uses a bilateral filter.

11. The method of claim 1, wherein the mapping function is

ƒ

(Δ

)=Δ

^β

,

wherein an amount of detail is inversely proportional to β

.

12. The method of claim 1, wherein the output image is

(1−

α

)Ĩ

+α

ƒ

({tilde over (D)})=ε

(Ĩ

),

wherein Ĩ

is the first intermediate image, α

is a blending factor during the merging, ƒ

is the mapping function, {tilde over (D)} is the enhanced detail layer, and ε

is an overall detail-enhancement operator.

13. The claim 1, further comprising:

applying, iteratively, hack projection after the sharping and merging steps to enforce consistency of the output image and second intermediate image with the input image.

14. The method of claim 1, wherein the input image is in color, and further comprising:

converting the input image to a YCbCr space, and applying the interpolation to a luminance channel of the YCbCr space.

15. The method of claim 1, wherein the reference patches in the combination are weighted.

16. The method of claim 1, wherein the reference patches are selected from same sized patches of the input image.

17. The method of claim 1, wherein the reference patches are selected from a dictionary of patches learned from the input image.

18. The method of claim 17, wherein the reference patches are selected by minimizing an error function.

19. The method of claim 18, wherein the error function is a weighted sum of a number of reference patches and a reconstruction error between a particular patch and a reconstruction of the particular patch by the reference patches.

20. The method of claim 18, wherein the error function is a weighted sum of absolute values of weights of the reference patches and a reconstruction error between a particular patch and a reconstruction of the particular patch by the reference patches.