System and Method of Image Upsampling

• US 20130121568A1
• Filed: 11/15/2011
• Published: 05/16/2013
• Est. Priority Date: 11/15/2011
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Krishnan, Shankar, Klosowski, James
• Granted Patent: US 8,571,309 B2
• Time in Patent Office: Days
• US Class Current: 382/162
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution

Abstract

A method includes receiving an image having a first resolution and generating an upsampled image having a second resolution based on the image. A multi-dimensional data structure corresponding to a multi-dimensional image space is generated from the upsampled image. Each node of the data structure is determined based on a weighted sum of values of one or more pixels in the upsampled image. Each of the one or more pixels corresponds to a pixel in the received image and is located within a region of the image space having a vertex defined by the node. A filter modifies the values of the nodes and a second upsampled image is generated based on the modified values of the nodes. Each pixel of the second upsampled image not corresponding to a pixel in the received image is determined based on a weighted sum of the modified values of one or more nodes.

Claims

1. A computer-implemented method, comprising:

receiving an image having a first resolution;

generating a first upsampled image based on the image, the first upsampled image having a second resolution that is greater than the first resolution;

generating a multi-dimensional data structure corresponding to a multi-dimensional image space based on the first upsampled image, wherein each node of the multi-dimensional data structure has a value that is determined based on a weighted sum of values of one or more pixels in the first upsampled image, wherein each of the one or more pixels corresponds to a pixel in the received image and is located within a region of the multi-dimensional image space that has a vertex defined by the node;

applying a filter to modify the values of the nodes of the multi-dimensional data structure; and

generating a second upsampled image based on the modified values of the nodes in the multi-dimensional data structure, wherein each pixel of the second upsampled image that does not correspond to a pixel in the received image has a pixel value that is determined based on a weighted sum of the modified values of one or more nodes of the multi-dimensional data structure.

2. The method of claim 1, wherein the first upsampled image is generated based on interpolating the image via at least one of bi-cubic interpolation, linear interpolation, and Gaussian interpolation.

3. The method of claim 1, wherein the second resolution is greater than the first resolution by an integer multiple.

4. The method of claim 1, wherein the multi-dimensional data structure is a five-dimensional data structure and wherein the multi-dimensional image space is a five-dimensional image space.

5. The method of claim 4, wherein the threshold region is a five-dimensional simplex having a vertex defined by the node.

6. The method of claim 4, wherein the five-dimensional image space comprises a first dimension along a first positional axis, a second dimension along a second positional axis, a third dimension corresponding to a first color, a fourth dimension corresponding to a second color, and a fifth dimension corresponding to a third color.

7. The method of claim 1, wherein the filter comprises a 1-2-1 filter.

8. The method of claim 1, further comprising de-blurring the second upsampled image using an image deconvolver.

9. The method of claim 8, wherein the image deconvolver is configured to apply an image deconvolution technique to an input image and wherein de-blurring the second upsampled image consists of executing a single iteration of the image deconvolution technique on the second upsampled image.

10. The method of claim 8, wherein a graphics processing unit generates the multi-dimensional data structure, applies the filter, generates the second upsampled image, and includes the image deconvolver.

11. The method of claim 1, wherein the received image corresponds to a first video frame, and further comprising:

receiving a second image having the first resolution, the second image corresponding to a second video frame;

generating a third upsampled image based on the second image, the third upsampled image having the second resolution;

generating a second multi-dimensional data structure based on the third upsampled image;

applying the filter to blur values of nodes in the second multi-dimensional data structure; and

generating a fourth upsampled image based at least in part on the blurred values of the nodes in the second multi-dimensional data structure.

12. An apparatus, comprising:

a processor;

an interpolator executable by the processor to generate a first upsampled image based on a received image, the received image having a first resolution and the first upsampled image having a second resolution that is greater than the first resolution;

a splatter executable by the processor to generate a multi-dimensional data structure corresponding to a multi-dimensional image space based on the first upsampled image, wherein each node of the multi-dimensional data structure has a value that is determined based on a weighted sum of values of one or more pixels in the first upsampled image, wherein each of the one or more pixels corresponds to a pixel in the received image and is located within a region of the multi-dimensional image space that has a vertex defined by the node;

a filter executable by the processor to modify the values of the nodes of the multi-dimensional data structure; and

a slicer executable by the processor to generate a second upsampled image based on the modified values of the nodes in the multi-dimensional data structure, wherein each pixel of the second upsampled image that does not correspond to a pixel in the received image is determined based on a weighted sum of the modified values of one or more nodes of the multi-dimensional data structure.

13. The apparatus of claim 12, further comprising an image deconvolver executable by the processor to de-blur the second upsampled image.

14. The apparatus of claim 13, wherein the image deconvolver de-blurs the second upsampled image by executing a single iteration of an image deconvolution technique.

15. The apparatus of claim 12, wherein the processor comprises a dedicated graphics processor.

16. A computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to:

receive an image having a first resolution;

generate a first upsampled image based on an interpolation of the image, the first upsampled image having a second resolution that is greater than the first resolution by an integer multiple;

generate a five-dimensional data structure corresponding to a five-dimensional image space based on the first upsampled image, wherein the five-dimensional image space includes a first dimension along a first positional axis, a second dimension along a second positional axis, a third dimension corresponding to a first color, a fourth dimension corresponding to a second color, and a fifth dimension corresponding to a third color, wherein each node of the five-dimensional data structure has a value that is determined based on a weighted sum of values of one or more pixels in the first upsampled image, wherein each of the one or more pixels corresponds to a pixel in the received image and is located within a five-dimensional simplex in the five-dimensional image space that has a vertex defined by the node;

apply a filter to modify the values of the nodes of the five-dimensional data structure; and

generate a second upsampled image based on the modified values of the nodes in the five-dimensional data structure, wherein each pixel of the second upsampled image that does not correspond to a pixel in the received image is determined based on a weighted sum of the modified values of one or more nodes of the five-dimensional data structure.

18. The computer-readable storage medium of claim 16, wherein the filter comprises a 1-2-1 filter.

19. The computer-readable storage medium of claim 16, further comprising de-blurring the second upsampled image using an image deconvolver.

20. The computer-readable storage medium of claim 19, wherein the image deconvolver is configured to iteratively apply an image deconvolution technique and wherein de-blurring the second upsampled image consists of executing a single iteration of the image deconvolution technique.

17. The method of computer-readable storage medium 16, wherein the first upsampled image is generated based on the image via at least one of bi-cubic interpolation, linear interpolation, and Gaussian interpolation.