High-Quality Upscaling of an Image Sequence

• US 20130071040A1
• Filed: 05/25/2012
• Published: 03/21/2013
• Est. Priority Date: 09/16/2011
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Jin, Hailin, Lin, Zhe, Cohen, Scott D., Ayvaci, Alper
• Granted Patent: US 9,087,390 B2
• Time in Patent Office: Days
• US Class Current: 382/254
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 3/4076 using the original low-resolution images to iteratively correct the high-resolution images
• CPC Class Codes: G06T 5/70 Denoising; Smoothing

Abstract

A method, system, and computer-readable storage medium are disclosed for upscaling an image sequence. An upsampled frame is generated based on an original frame in an original image sequence comprising a plurality of frames. A smoothed image sequence is generated based on the original image sequence. A plurality of patches are determined in the upsampled frame. Each patch comprises a subset of image data in the upsampled frame. Locations of a plurality of corresponding patches are determined in a neighboring set of the plurality of frames in the smoothed image sequence. A plurality of high-frequency patches are generated. Each high-frequency patch is based on image data at the locations of the corresponding patches in the original image sequence. The plurality of high-frequency patches are added to the upsampled frame to generate a high-quality upscaled frame.

Claims

1. A computer-implemented method, comprising:

generating an upsampled frame based on an original frame in an original image sequence, wherein the original image sequence comprises a plurality of frames;

generating a smoothed image sequence based on the original image sequence;

determining a plurality of patches in the upsampled frame, wherein each of the plurality of patches comprises a subset of image data in the upsampled frame;

determining locations of a plurality of corresponding patches in a neighboring set of the plurality of frames in the smoothed image sequence;

generating a plurality of high-frequency patches, wherein each high-frequency patch is based on image data at the locations of the plurality of corresponding patches in the original image sequence; and

adding the plurality of high-frequency patches to the upsampled frame.

2. The method as recited in claim 1, further comprising:

determining an optical flow in the original image sequence;

wherein the locations of the plurality of corresponding patches are determined based on the optical flow.

3. The method as recited in claim 1, wherein determining the locations of the plurality of corresponding patches comprises searching a local window in each of the neighboring set of the plurality of frames in the smoothed image sequence.

4. The method as recited in claim 1, wherein determining the locations of the plurality of corresponding patches comprises excluding a patch comprising an occluded pixel.

5. The method as recited in claim 1, wherein the locations of the plurality of corresponding patches are determined based on an occlusion map.

6. The method as recited in claim 1, wherein adding the plurality of high-frequency patches to the upsampled frame comprises averaging values for overlapping pixels.

7. The method as recited in claim 1, further comprising:

denoising the image sequence prior to generating the upsampled frame.

8. A system, comprising:

at least one processor; and

a memory coupled to the at least one processor, wherein the memory stores program instructions, wherein the program instructions are executable by the at least one processor to;

generate an upsampled frame based on an original frame in an original image sequence, wherein the original image sequence comprises a plurality of frames;

generate a smoothed image sequence based on the original image sequence;

determine a plurality of patches in the upsampled frame, wherein each of the plurality of patches comprises a subset of image data in the upsampled frame;

determine locations of a plurality of corresponding patches in a neighboring set of the plurality of frames in the smoothed image sequence;

generate a plurality of high-frequency patches, wherein each high-frequency patch is based on image data at the locations of the plurality of corresponding patches in the original image sequence; and

add the plurality of high-frequency patches to the upsampled frame.

9. The system as recited in claim 8, wherein the program instructions are executable by the at least one processor to:

determining an optical flow in the original image sequence;

wherein the locations of the plurality of corresponding patches are determined based on the optical flow.

10. The system as recited in claim 8, wherein, in determining the locations of the plurality of corresponding patches, the program instructions are executable by the at least one processor to search a local window in each of the neighboring set of the plurality of frames in the smoothed image sequence.

11. The system as recited in claim 8, wherein, in determining the locations of the plurality of corresponding patches, the program instructions are executable by the at least one processor to exclude a patch comprising an occluded pixel.

12. The system as recited in claim 8, wherein the locations of the plurality of corresponding patches are determined based on an occlusion map.

13. The system as recited in claim 8, wherein, in adding the plurality of high-frequency patches to the upsampled frame, the program instructions are executable by the at least one processor to average values for overlapping pixels.

14. The system as recited in claim 8, wherein the program instructions are executable by the at least one processor to:

denoising the image sequence prior to generating the upsampled frame.

15. A computer-readable storage medium storing program instructions computer-executable to perform:

generating an upsampled frame based on an original frame in an original image sequence, wherein the original image sequence comprises a plurality of frames;

generating a smoothed image sequence based on the original image sequence;

determining a plurality of patches in the upsampled frame, wherein each of the plurality of patches comprises a subset of image data in the upsampled frame;

determining locations of a plurality of corresponding patches in a neighboring set of the plurality of frames in the smoothed image sequence;

generating a plurality of high-frequency patches, wherein each high-frequency patch is based on image data at the locations of the plurality of corresponding patches in the original image sequence; and

adding the plurality of high-frequency patches to the upsampled frame.

16. The computer-readable storage medium as recited in claim 15, further comprising:

determining an optical flow in the original image sequence;

wherein the locations of the plurality of corresponding patches are determined based on the optical flow.

17. The computer-readable storage medium as recited in claim 15, wherein determining the locations of the plurality of corresponding patches comprises searching a local window in each of the neighboring set of the plurality of frames in the smoothed image sequence.

18. The computer-readable storage medium as recited in claim 15, wherein determining the locations of the plurality of corresponding patches comprises excluding a patch comprising an occluded pixel.

19. The computer-readable storage medium as recited in claim 15, wherein adding the plurality of high-frequency patches to the upsampled frame comprises averaging values for overlapping pixels.

20. The computer-readable storage medium as recited in claim 15, further comprising:

denoising the image sequence prior to generating the upsampled frame.