Techniques and systems for developing high-resolution imagery

• US 7,260,274 B2
• Filed: 11/30/2001
• Issued: 08/21/2007
• Est. Priority Date: 12/01/2000
• Status: Expired due to Fees

• Current Assignee: IMAX Corporation
• Original Assignee: IMAX Corporation
• Inventors: Hanna, Keith J., Kumar, Rakesh, Guo, Yanlin, Bergen, James R., Sawhney, Harpreet S.
• Primary Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US10/433,338
• Publication Number:
  US 20040151401A1
• Time in Patent Office: 2,090 Days
• Field of Search: 382/154, 382/254, 382294-300, 382/107, 382/318, 382/151, 382/176, 382/174, 382/240, 382/278, 382/284, 382/312, 345419-420, 345/613, 348/42, 348/47, 358/1.2, 375/240.19
• US Class Current: 382/284
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/403 Edge-driven scaling; Edge-based scaling
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Systems and techniques for synthesizing high-resolution images are described. Using as input hybrid high- and low-resolution images (110, 120), processing techniques including three-dimensional image warp-based (160) rendering may assist in developing the high-resolution output images. Enhanced results may in some instances be obtained by creating a high-resolution images of one member of a pair of stereo images by utilizing multiple frames of the other member of the pair.

Claims

1. A method for synthesizing a high resolution image comprising:

receiving a sequence pair of images comprising a first image of a low resolution taken from a first camera and a second image of a high resolution taken from a second camera;

receiving a plurality of images temporally adjacent to the second image, the plurality of images being of a high resolution and taken from the second camera;

computing stereo correspondence maps between the first image and the second image;

computing temporal correspondence maps between the first image and each of the plurality of images; and

computing a composite high resolution version of the first image using the stereo correspondence maps and the temporal correspondence maps.

2. The method of claim 1, wherein the first image comprises a coordinate system, the method further comprising:

warping the second image to the coordinate system;

computing stereo alignment maps between the warped second image and the first image;

warping the plurality of images to the coordinate system;

computing motion alignment maps between of the warped plurality of images and the first image;

producing a composite correlation map by combining the stereo alignment maps and the motion alignment maps; and

projecting the composite correlation map to the high resolution of the second image.

3. The method of claim 2, wherein the composite high resolution version of the first image comprises mis-aligned regions, the method further comprising:

filling the mis-aligned regions based on the composite correlation map.

4. The method of claim 3, further comprising filling the mis-aligned regions with a color-corrected, enhanced, up-sampled first image.

5. The method of claim 4, further comprising color-correcting the first image using a local adaptive color correction technique.

6. The method of claim 1, further comprising:

computing a new correlation mask for each composite image by imposing temporal consistency; and

filling composite images using the new correlation masks.

7. A method for synthesizing a high resolution image comprising:

receiving a sequence pair comprising a first image of a low resolution taken from a first camera and a second image of a high resolution taken from a second camera;

receiving a third image and a fourth image temporally adjacent to the second image, the third and fourth image being of a high resolution and taken from the second camera;

up-sampling the first image to full resolution;

applying image enhancement to the up-sampled image;

computing a planar parametric motion estimation for a specified planar region in the images to determine a planar alignment between the first image and the second image;

using the planar alignment in computing a correspondence map between the first image and the second image using plane-plus-parallax alignment at the low resolution of the first image;

establishing a correspondence between the first image, the second image, the third image and the fourth image that is displaced in rime by using optical flow;

generating warped images from the second image, the third image and the fourth image using the correspondence map;

computing stereo alignment maps between the warped images and the first image for color channels at a pyramid level 2;

computing motion alignment maps between the third and fourth image and the first image for the color channels at pyramid level 2;

combining all maps to produce a composite correlation map for each of the color channels at pyramid level 2;

projecting the composite correlation map to pyramid level 0;

combining all warped images to create a final composite image at pyramid level 2 and pyramid level 0;

computing a correlation mask between the composite image and the first image at pyramid level 2 and project the mask to pyramid level 0; and

fill in mis-aligned regions in the composite image with the color corrected, enhanced, and up-sampled first image.