System for creating a high resolution image from a sequence of lower resolution motion images

• US 5,696,848 A
• Filed: 01/17/1996
• Issued: 12/09/1997
• Est. Priority Date: 03/09/1995
• Status: Expired due to Term

• Current Assignee: Eastman Kodak Company
• Original Assignee: Eastman Kodak Company
• Inventors: Sezan, M. Ibrahim, Eren, P. Erhan, Patti, Andrew J.
• Primary Examiner(s): Couso, Jose L.
• Assistant Examiner(s): Bella, Matthew C.
• Application Number:
  US08/588,004
• Time in Patent Office: 692 Days
• Field of Search: 382/254, 382/298, 382/299, 382/300, 382/308
• US Class Current: 382/254
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 5/14 Picture signal circuitry for video frequency region cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 7/0125 one of the standards being a high definition standard
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

A robust system, adaptive to motion estimation accuracy, for creating a high resolution image from a sequence of lower resolution motion images produces a mapping transformation for each low resolution image to map pixels in each low resolution image into locations in the high resolution image. A combined point spread function (PSF) is computed for each pixel in each lower resolution image employing the mapping transformations provided that they describe accurate motion vectors. The high resolution image is generated from the lower resolution images employing the combined PSF'"'"'s by projection onto convex sets (POCS), where sets and associated projections are defined only for those pixels whose motion vector estimates are accurate.

Claims

1. A method for creating a high resolution image from a sequence of lower resolution motion images, comprising the steps of:

a. producing a mapping transformation for each lower resolution image to map pixels in each lower resolution image into locations in the high resolution image;

b. computing a combined point spread function (combined PSF) for each pixel in each lower resolution image employing the mapping transformations, by;

i. calculating an effective sensor aperture relative to the high resolution image for each pixel in each lower resolution image employing the mapping transformations;

ii. calculating PSF'"'"'s for the effective sensor apertures;

iii. defining an optical PSF; and

iv. combining the calculated PSF for each pixel with the optical PSF to produce the combined PSF for each pixel;

c. generating the high resolution image from the lower resolution images employing the combined PSF by projection onto convex sets (POCS); and

d. displaying the high resolution image.

2. The method claimed in claim 1, wherein the step of producing a mapping transformation includes:

a. selecting one of the lower resolution images as a reference image;

b. estimating a mapping transformation describing the relative motion at each pixel between the reference lower resolution image and each other lower resolution image;

c. testing the validity of the estimated mapping transformation for each other lower resolution image pixel and generating a motion accuracy map representing valid mapping transformations for each other lower resolution image; and

d. scaling each valid mapping transformation from the lower resolution images to the high resolution image.

3. The method claimed in claim 1, wherein the step of generating the high resolution image by POCS includes:

a. interpolating one of the lower resolution images to the number of pixels in the high resolution image to produce an estimate of the high resolution image; and

b. for each pixel in each low resolution image having a valid mapping transformation, refining the estimate of the high resolution image by,

i. selecting a pixel in one of the lower resolution images,

ii. producing a calculated pixel value from the high resolution image by applying the combined PSF for the selected pixel to the current estimate of the high resolution image, and

iii. forming a difference between the selected pixel value and the calculated pixel value and f the magnitude of the difference is greater than a predetermined threshold, back projecting the difference into the current estimate of the high resolution image to produce a refined estimate of the high resolution image;

c. applying an amplitude constraint to the pixel values of the refined estimate of the high resolution image; and

d. repeating steps b and c until a stopping criterion is satisfied.

4. A method for producing a high resolution video sequence from a sequence of lower resolution video images, comprising applying the method claimed in claim 1 a plurality of times to the sequence of lower resolution video images to produce a video sequence of high resolution images.

6. The method claimed in claim 2, wherein the step of testing the validity of the estimated mapping transformation for each other lower resolution image pixel, further comprises the steps of:

a. determining a threshold;

b. computing a summed absolute difference image;

c. comparing the summed absolute difference image with the threshold to produce an initial motion accuracy map; and

d. employing morphological erosion and dilation to the initial motion accuracy map to produce a final motion accuracy map.

7. The method claimed in claim 6, wherein the step of determining a threshold, further comprises the steps of:

a. estimating a standard deviation image by computing a sample standard deviation at every original pixel location of the other lower resolution images;

b. quantizing the standard deviation image to two levels; and

c. scaling the quantized two level image by a predetermined maximum threshold.

5. Apparatus for creating a high resolution image from a sequence of lower resolution motion images, comprising:

a. a source for producing a sequence of lower resolution motion images;

b. an image processor for receiving the sequence of lower resolution images and creating the high resolution image, including;

i. means for producing a mapping transformation for each low resolution image to map pixels in each low resolution image into locations in the high resolution image;

ii. means for computing a combined point spread function (combined PSF) for each pixel in each lower resolution image employing the mapping transformations, the means for computing a combined point spread function including means for calculating an effective sensor aperture relative to the high resolution image for each pixel in each lower resolution image employing the mapping transformations, means for calculating PSF'"'"'s for the effective sensor apertures, means for defining an optical PSF, and means for combining the calculated PSF for each pixel with the optical PSF to produce the combined PSF for each pixel;

iii. means for generating the high resolution image from the lower resolution images employing the combined PSF by projection onto convex sets (POCS); and

c. a display device for displaying the high resolution image.

8. A computer program product, comprising a computer readable storage medium having a computer program stored thereon for creating a high resolution image from a sequence of lower resolution motion images, by the steps of:

a. producing a mapping transformation for each lower resolution image to map pixels in each lower resolution image into locations in the high resolution image;

b. computing a combined point spread function (combined PSF) for each pixel in each lower resolution image employing the mapping transformations, by

i. calculating an effective sensor aperture relative to the high resolution image for each pixel in each lower resolution image employing the mapping transformations;

ii. calculating PSF'"'"'s for the effective sensor apertures;

iii. defining an optical PSF; and

iv. combining the calculated PSF for each pixel with the optical PSF to produce the combined PSF for each pixel;

c. generating the high resolution image from the lower resolution images employing the combined PSF by projection onto convex sets (POCS); and

d. displaying the high resolution image.