Techniques and systems for developing high-resolution imagery

• US 20040151401A1
• Filed: 03/03/2004
• Published: 08/05/2004
• Est. Priority Date: 12/01/2000
• Status: Active Grant

• Current Assignee: IMAX Corporation
• Original Assignee: IMAX Corporation
• Inventors: Hanna, Keith J, Bergen, James R, Kumar, Rakesh, Guo, Yanlin, Sawhney, Harpreet S.
• Granted Patent: US 7,260,274 B2
• Time in Patent Office: Days
• US Class Current: 382/299
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/403 Edge-driven scaling; Edge-based scaling
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Systems and techniques for synthesizing high-resolution images are described. Using as input hybrid high- and low-resolution images (110, 120), processing techniques including three-dimensional image warp-based (160) rendering may assist in developing the high-resolution output images. Enhanced results may in some instances be obtained by creating a high-resolution images of one member of a pair of stereo images by utilizing multiple frames of the other member of the pair.

Claims

1. A method for synthesizing a high resolution image comprising:

receiving a sequence pair of images comprising a first image of a low resolution taken from a first camera and a second image of a high resolution taken from a second camera;

receiving a third image and a fourth image temporally adjacent to the second image, the third and fourth image being of a high resolution and taken from the second camera;

developing correspondence maps between the first image and the second image by using a surface in the images to establish a reference frame between the two images and computing a three dimensional shape with respect to the reference plane using the third image and the fourth image; and

computing high resolution first image using the correspondence maps.

2. An apparatus for synthesizing a high resolution image comprising:

means for receiving a sequence pair of images comprising a first image of a low resolution taken from a first camera and a second image of a high resolution taken from a second camera;

means for receiving a third image and a fourth image temporally adjacent to the second image, the third and fourth image being of a high resolution and taken from the second camera;

means for developing correspondence maps between the first image and the second image by using a surface in the images to establish a reference frame between the two images and computing a three dimensional shape with respect to the reference plane using the third image and the fourth image; and

means for computing high resolution first image using the correspondence maps.

3. A method for synthesizing a high resolution image comprising:

receiving a sequence pair of images comprising a first image of a low resolution taken from a first camera and a second image of a high resolution taken from a second camera;

receiving a third image and a fourth image temporally adjacent to the second image, the third and fourth image being of a high resolution and taken from the second camera;

up-sampling the first image to full resolution;

applying image enhancement to the up-sampled image;

computing a planar parametric motion estimation for a specified planar region in the images to determine a planar alignment between the first image and the second image;

using the planar alignment in computing a correspondence map between the first image and the second image using plane-plus-parallax alignment at the low resolution of the first image;

establishing a correspondence between the first image, the second image, the third image and the fourth image that is displaced in time by using optical flow;

generating warped images from the second image, the third image and the fourth image using the correspondence map; and

computing high resolution first image using the correspondence map and the warped images.

4. The method of claim 3 wherein the specified planar region is a rectangular window in a coordinate system of the first image.

5. The method of claim 3 wherein the correspondence map between the first and second images is parameterized by multiple parameters for the relative pose between the cameras and by one pixel for a relative shape.

6. The method of claim 3 further comprising:

correcting errors in the correspondence map by detecting misaligned regions, removing the detected regions, and blending in information from lower-resolution imagery, adjacent regions of the image, or adjacent temporal frames.

7. A method for synthesizing a high resolution image comprising:

receiving a sequence pair comprising a first image of a low resolution taken from a first camera and a second image of a high resolution taken from a second camera, receiving a third image and a fourth image temporally adjacent to the second image, the third and fourth image being of a high resolution and taken from the second camera;

up-sampling the first image to full resolution;

applying image enhancement to the up-sampled image;

computing a planar parametric motion estimation for a specified planar region in the images to determine a planar alignment between the first image and the second image;

using the planar alignment in computing a correspondence map between the first image and the second image using plane-plus-parallax alignment at the low resolution of the first image;

establishing a correspondence between the first image, the second image, the third image and the fourth image that is displaced in time by using optical flow;

generating warped images from the second image, the third image and the fourth image using the correspondence map;

computing stereo alignment maps between the warped images and the first image for color channels at a pyramid level 2;

computing motion alignment maps between the third and fourth image and the first image for the color channels at pyramid level 2;

combining all maps to produce a composite correlation map for each of the color channels at pyramid level 2;

projecting the composite correlation map to pyramid level 0;

combining all warped images to create a final composite image at pyramid level 2 and pyramid level 0;

computing a correlation mask between the composite image and the first image at pyramid level 2 and project the mask to pyramid level 0; and

fill in mis-aligned regions in the composite image with the color corrected, enhanced, and up-sampled first image.