Resolution improvement from multiple images of a scene containing motion at fractional pixel values

• US 6,285,804 B1
• Filed: 12/21/1998
• Issued: 09/04/2001
• Est. Priority Date: 12/21/1998
• Status: Expired due to Term

• Current Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Sharp Laboratories of America Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Crinon, Regis J., Sezan, Muhammed Ibrahim
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US09/217,150
• Time in Patent Office: 988 Days
• Field of Search: 348/699, 382/299, 382/300, 382/236, 382/191, 382/107
• US Class Current: 382/299
• CPC Class Codes: G06T 3/4069 by subpixel displacements

Abstract

A high-resolution image is derived from multiple low resolution images each having an array of low-resolution pixels. Motion vectors are derived at each of the unknown high-resolution grid points for each of the multiple low-resolution images. One motion vector is generated at each one of the high-resolution grid locations for each one of the multiple images. The motion vectors at the high-resolution grid points associate high-resolution grid points, whose values are unknown, with inter-pixel positions on the associated low-resolution images. Low-resolution pixels are identified that have the closest distance to each inter-pixel position. One or several of the identified low-resolution pixels having the shortest distance are used to determine the pixel value at each one of the high-resolution grid points. Pixel intensity values are then mapped into the high-resolution grid points according to the selected low-resolution pixels.

Claims

1. A method for generating a high-resolution image having an array of proportionally spaced high-resolution grid points derived from multiple low resolution images each having an array of low resolution pixels, comprising:

deriving motion vectors that each originate from the proportionally spaced high-resolution grid points in the high-resolution image, the motion vectors each associated with one of the multiple low resolution images;

associating the high-resolution grid points with inter-pixel positions on the associated low-resolution images using the motion vectors;

identifying one or more of the low-resolution pixels on each one of the low-resolution images having the closest spatial distance to the inter-pixel positions;

selecting one or more of the identified low-resolution pixels for each one of the high-resolution grid points having a shortest distance; and

mapping pixel intensity values into the high-resolution grid points according to the selected low-resolution pixels.

2. A method according to claim 1 wherein mapping the pixel intensity values comprises interpolating the intensity sample values at the mapped inter-pixel positions from the neighboring low-resolution pixels.

3. A method according to claim 1 wherein mapping the pixel intensity values comprise mapping the intensity values of the pixel closest to the mapped inter-pixel positions directly back as the pixel value at the high-resolution grid locations.

4. A method according to claim 1 wherein the motion vectors at the high-resolution grid points are estimated according to one of the following:

using a median vector value of neighboring motion vectors for existing low-resolution pixels;

using a motion model, such as a translational, an affine or a perspective model;

fitting the motion model to pixel motion information belonging to the low-resolution pixels surrounding the high-resolution grid points; and

estimating the motion vectors at the high resolution grid points from the motion model.

5. A method according to claim 4 including generating one motion vector at each one of the high-resolution grid locations for each one of the multiple low-resolution images.

6. A method according to claim 5 including the following:

selecting a distance threshold value; and

using a spatial and/or temporal smart interpolation or an edge adaptive interpolation for deriving pixel values at the high-resolution grid locations when the pixel having the shortest distance to the mapped inter-pixel position is greater than the distance threshold value.

7. A method according to claim 5 including computing a local motion compensation error at every high resolution sampling grid point according to a sum of sample intensity differences of pixel values in the low resolution reference image and in the low resolution target image and using a spatial and/or temporal smart interpolation or an edge adaptive interpolation for deriving pixel values at the high-resolution grid locations when the motion compensation error is high.

8. A method according to claim 5 including the following:

generating multiple color channels for each of the multiple images; and

mapping the pixel intensity values for the high-resolution grid locations according to a common motion estimation model derived from one or several color channels or luminance.

9. A system for generating a high-resolution image, comprising:

a processor receiving multiple low-resolution images each having an associated motion information field that identifies relative motion occurring while generating the multiple low-resolution images; and

the processor generating a high-resolution image from pixels in the multiple low resolution images by mapping motion vectors originating from the high-resolution grid points in the high-resolution image to inter-pixel locations in the low-resolution images and identifying which of the pixels in the multiple low-resolution images are most closely located to the inter-pixel locations and then mapping image intensity values into the high-resolution grid points according to the identified pixels in the low-resolution images.

10. A system according to claim 9 wherein the processor selectively varies a resolution factor of the high-resolution image, the processor varying the number of low-resolution images used for generating the high-resolution image according to the resolution factor.

11. A system according to claim 9 including deriving a median motion vector at each high-resolution grid point for each one of the low-resolution images by taking a vector median of motion vectors at existing low-resolution pixel locations in a reference one of the multiple low-resolution images.

12. A system according to claim 9 wherein the processor locates the pixel in each of the low-resolution images that is located closest to the inter-pixel locations and selects one or several of the located pixels that have the shortest distances to the inter-pixel locations for generating the pixel values at the high-resolution grid points.