Interpolation of a sequence of images using motion analysis

• US 6,570,624 B2
• Filed: 04/20/2001
• Issued: 05/27/2003
• Est. Priority Date: 09/08/2000
• Status: Expired due to Term

• Current Assignee: Avid Technology Incorporated
• Original Assignee: Avid Technology Incorporated
• Inventors: Laird, Michael, Fayan, Randy M., Dickie, Garth A., Fasciano, Peter J., Cornog, Katherine H., Gonsalves, Robert A.
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US09/839,160
• Publication Number:
  US 20030035592A1
• Time in Patent Office: 767 Days
• Field of Search: 348/448, 348/451, 348/452, 348/446, 382/300, 382/236, 382/276
• US Class Current: 348/446
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 2210/44 Morphing
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: H04N 5/147 Scene change detection
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

Two images are analyzed to compute a set of motion vectors that describes motion between the first and second images. A motion vector is computed for each pixel in an image at a time between the first and second images. This set of motion vectors may be defined at any time between the first and second images, such as the midpoint. The motion vectors may be computed using any of several techniques. An example technique is based on the constant brightness constraint, also referred to as optical flow. Each vector is specified at a pixel center in an image defined at the time between the first and second images. The vectors may point to points in the first and second images that are not on pixel centers. The motion vectors are used to warp the first and second images to a point in time of an output image between the first and second images using a factor that represents the time between the first and second image at which the output image occurs. The warped images are then blended using this factor to obtain the output image at the desired point in time between the first and second images. The point in time at which the output image occurs may be different from the time at which the motion vectors are determined. The same motion vectors may be used to determine two or more output images at different times between the first and second images. The images may be warped using a technique in which many small triangles are defined in an image corresponding in time to the point in time between the first and second images at which the motion vectors are determined. A transform for each small triangle from the point in time at which the motion vectors are determined to the desired interpolated image time is determined, e.g., the triangle is warped using the motion vectors associated with its vertices. For each pixel in each triangle in the output image, corresponding points in the first and second images are determined, and the first and second images are spatially sampled at these points. These samples for each pixel are combined to produce a value for that pixel in the output image. A motion vector map also may be generated between two fields of opposite field sense. An offset introduced by such calculation may be removed. During warping or other operation using the adjusted vector map, when sampling a field of one field sense to generate a field of an opposite field sense, that offset is reintroduced by translating the sampling region.

Claims

1. A method for processing two fields of interlaced video, comprising:

computing motion vectors describing motion of image characteristics from a field to another field of opposite sense;

removing from the motion vectors an offset corresponding to one half of a line and having a sign according to an orientation of the y-axis of the image space and which field includes the top line;

using the motion vectors to generate a sampling region at a desired output time;

transforming the sampling region using the motion vectors to a sample time at one of the fields;

transforming the sampling region using the motion vectors to a sample time at the other of the fields;

determining a field sense of an output field to be generated at the desired output time;

translating the transformed sampling region for the field with a field sense opposite the field sense of the output field by an offset of one half of a line and having a sign determined by an orientation of the y-axis of the image space and which field includes the top line; and

generating the output field using the transformed and translated sampling regions and the input fields.

2. The method of claim 1, further comprising:

wherein the desired output time is the time of one of the fields;

combining the generated output field with the field at the desired output time to generate a progressive image.

3. The method of claim 2, further comprising:

performing an effect on the progressive image.

4. The method of claim 3, further comprising:

vertically decimating the progressive image with the effect to produce a field at the desired output time.

5. A method for processing two fields of interlaced video, comprising:

computing motion vectors describing motion of image characteristics from a field to another field of opposite sense;

removing from the motion vectors an offset corresponding to one half of a line and having a sign according to an orientation of the y-axis of the image space and which field includes the top line;

selecting a time of one of the fields as a desired output time;

transforming a sampling region specified at the selected time using the motion vectors to a sample time at the field that is being warped;

translating the transformed sampling region by an offset of one half of a line and having a sign determined by an orientation of the y-axis of the image space and which field includes the top line; and

generating the output field at the desired output time using the transformed and translated sampling region and the field that is being warped.

6. The method of claim 5, further comprising:

combining the generated output field with the field at the desired output time to generate a progressive image.

7. The method of claim 6, further comprising:

performing an effect on the progressive image.

8. The method of claim 7, further comprising:

vertically decimating the progressive image with the effect to produce a field at the desired output time.

9. The method of claim 6, further comprising:

exporting the progressive image to a file.

10. The method of claim 6, further comprising:

displaying the progressive image as a still frame.

11. The method of claim 10, wherein displaying the progressive image as a still frame includes displaying the progressive image in a freeze frame effect.

12. The method of claim 10, wherein displaying the progressive image as a still frame includes displaying the progressive image upon stopping playback.

13. The method of claim 5, further comprising:

warping another field of the opposite sense to the desired output time; and

blending the two warped fields to produce a field at the desired output time.