Interpolation of a sequence of images using motion analysis
First Claim
1. A method for generating an output image associated with a point in time between a first image and a second image, comprising:
- determining a motion vector for each pixel in an image at a map time between the first image and the second image, wherein the map time is different from the point in time of the output image, wherein the motion vector describes motion of a pixel of the image at the map time to a first point in the first image and a second point in the second image;
calculating a factor that represents the point in time between the first image and the second image at which the output image occurs;
warping the first image according to the determined motion vectors and the factor;
warping the second image according to the determined motion vectors and the factor; and
blending the warped first image and the warped second image according to the factor to obtain the output image.
9 Assignments
0 Petitions
Accused Products
Abstract
Two images are analyzed to compute a set of motion vectors that describes motion between the first and second images. A motion vector is computed for each pixel in an image at a time between the first and second images. This set of motion vectors may be defined at any time between the first and second images, such as the midpoint. The motion vectors may be computed using any of several techniques. An example technique is based on the constant brightness constraint, also referred to as optical flow. Each vector is specified at a pixel center in an image defined at the time between the first and second images. The vectors may point to points in the first and second images that are not on pixel centers. The motion vectors are used to warp the first and second images to a point in time of an output image between the first and second images using a factor that represents the time between the first and second image at which the output image occurs. The warped images are then blended using this factor to obtain the output image at the desired point in time between the first and second images. The point in time at which the output image occurs may be different from the time at which the motion vectors are determined. The same motion vectors may be used to determine two or more output images at different times between the first and second images. The images may be warped using a technique in which many small triangles are defined in an image corresponding in time to the point in time between the first and second images at which the motion vectors are determined. A transform for each small triangle from the point in time at which the motion vectors are determined to the desired interpolated image time is determined, e.g., the triangle is warped using the motion vectors associated with its vertices. For each pixel in each triangle in the output image, corresponding points in the first and second images are determined, and the first and second images are spatially sampled at these points. These samples for each pixel are combined to produce a value for that pixel in the output image. A motion vector map also may be generated between two fields of opposite field sense. An offset introduced by such calculation may be removed. During warping or other operation using the adjusted vector map, when sampling a field of one field sense to generate a field of an opposite field sense, that offset is reintroduced by translating the sampling region.
122 Citations
52 Claims
-
1. A method for generating an output image associated with a point in time between a first image and a second image, comprising:
-
determining a motion vector for each pixel in an image at a map time between the first image and the second image, wherein the map time is different from the point in time of the output image, wherein the motion vector describes motion of a pixel of the image at the map time to a first point in the first image and a second point in the second image;
calculating a factor that represents the point in time between the first image and the second image at which the output image occurs;
warping the first image according to the determined motion vectors and the factor;
warping the second image according to the determined motion vectors and the factor; and
blending the warped first image and the warped second image according to the factor to obtain the output image. - View Dependent Claims (2, 3, 4, 5, 6, 37, 38, 39)
-
-
7. A method for generating a plurality of output images, wherein each output image is associated with a different point in time between a first image and a second image, the method comprising:
-
determining a motion vector for each pixel in an image at a map time between the first image and the second image, wherein the motion vector describes motion of a pixel of the image at the map time to a first point in the first image and a second point in the second image;
for each output image, calculating a factor that represents the point in time between the first image and the second image at which the output image occurs;
for each output image, warping the first image according to the determined motion vectors and the factor for the output image;
for each output image, warping the second image according to the determined motion vectors and the factor for the output image; and
for each output image, blending the warped first image and the warped second image according to the factor for the output image. - View Dependent Claims (8, 9, 10, 11, 12)
-
-
13. A method for generating a plurality of output images, wherein each output image is associated with a different point in time between a first image of a first sequence of one or more images and a second image of a second sequence of one or more images, the method comprising:
-
for each output image, selecting a pair of a first image from the first sequence and a second image from the second sequence;
for each selected pair of first and second images, determining a motion vector for each pixel in an image at a map time between the first image and the second image, wherein the motion vector describes motion of a pixel of the image at the map time to a first point in the first image and a second point in the second image;
for each output image, calculating a factor that represents the point in time, between the first and second images selected for the output image, at which the output image occurs;
for each output image, warping the first image selected for the output image according to the factor for the output image and the motion vectors determined for the first and second images selected for the output image;
for each output image, warping the second image selected for the output image according to the factor for the output image and the motion vectors determined for the first and second images selected for the output image; and
for each output image, blending the warped first image and the warped second image according to the factor for the output image. - View Dependent Claims (14, 15, 16)
-
-
17. A method for generating a transition of a plurality of output images from a first sequence of images to a second sequence of images wherein an image at an end of the first sequence is not contiguous with an image at a beginning of the second sequence, the method comprising:
-
for each output image, selecting a pair of a first image from the first sequence and a second image from the second sequence such that the output image has a point in time between the first image and the second image in the transition;
for each selected pair of first and second images, determining a set of motion vectors that describes motion between the first and second images;
for each output image, calculating a factor that represents the point in time, between the first and second images selected for the output image, at which the output image occurs;
for each output image, performing motion compensated interpolation to generate the output image according to the determined set of motion vectors and the calculated factor. - View Dependent Claims (18, 19)
-
-
20. A method for processing a jump cut from a first image at an end of a first segment of sequence of images and corresponding audio and a second image at a beginning of a second segment in the sequence of images and corresponding audio, comprising:
-
processing the corresponding audio to identify an audio break between the audio corresponding to the first segment and the audio corresponding to the second segment;
determining a set of motion vectors that describes motion between the first and second images; and
performing motion compensated interpolation to generate one or more images between the first image and the second image according to the determined set of motion vectors at a point in time corresponding to the audio break. - View Dependent Claims (21, 22, 23)
-
-
24. A method for warping a first image and a second image to obtain an output image at an output time between the first image and the second image, comprising:
-
determining a set of motion vectors at a map time and that describes motion between the first and second images;
determining a primary transform for each triangle in a set of triangles, defined in an image at the map time, from the map time to the output time using the determined set of motion vectors;
for each triangle, identifying any pixels in the output image that are contained within the triangle using the primary transform;
determining a first transform for each triangle in the set of triangles from the output time to a time of the first image;
for each pixel in each triangle at the output time, identifying a point in the first image using the first transform and spatially sampling the first image around the point;
determining a second transform for each triangle in the set of triangles from the output time to a time of the second image;
for each pixel in each triangle at the output time, identifying a point in the second image using the second transform and spatially sampling the second image around the point; and
for each pixel in each triangle at the output time, combining the spatially sampled first image and the spatially sampled second image to obtain a value for the pixel in the output image. - View Dependent Claims (25, 26)
-
-
27. A method for warping a first image and a second image to obtain an output image at an output time between the first image and the second image, comprising:
-
determining a set of motion vectors at a map time and that describes motion between the first and second images;
determining a primary transform for each triangle in a set of triangles, defined in an image at the map time, from the map time to the output time using the determined set of motion vectors;
for each triangle, identifying any pixels in the output image that are contained within the triangle at the output time using the primary transform;
for each pixel in each triangle at the output time, spatially sampling the first image and the second image at points corresponding to the pixel and combining the spatially sampled first image and the spatially sampled second image to obtain a value for the pixel in the output image. - View Dependent Claims (28, 29)
-
-
30. A method for changing duration of an input sequence of images with associated audio, wherein the input sequence of images and associated audio has a duration, comprising:
receiving an indication of a selection of an operation by an operator indicative of a desired duration of an output sequence of images, and, in response to the received indication;
selecting a first image and a second image in the sequence of images;
determining a set of motion vectors that describes motion between the first and second images;
performing motion compensated interpolation to generate one or more images between the first image and the second image according to the determined set of motion vectors;
repeating the selecting, determining and performing steps for multiple pairs of first and second images in the sequence of images to provide the output sequence of images; and
adjusting the duration of the associated audio to retain synchronization with the output sequence of images. - View Dependent Claims (31, 32, 33)
-
34. A method for performing color correction, comprising:
-
generating a first color histogram from first image from a first sequence of images;
generating a second color histogram from a second image from a second sequence of images;
determining a set of motion vectors from the first and second color histograms that describes motion between the first color histogram and the second color histogram;
generating a table of color correction values from the set of motion vectors; and
applying the table of color correction values to a sequence of images.
-
-
35. A method for reducing artifacts in an image created using motion compensated interpolation of a first image and a second image, comprising:
-
determining a set of motion vectors that describes motion between the first and second images;
identifying a foreground region and a background region in the first and second images;
performing tracking on at least one of the foreground region and the background region to determine a motion model for the tracked region;
changing the set of motion vectors corresponding to the tracked region according to the motion model for the tracked region; and
performing motion compensated interpolation to generate one or more images between the first image and the second image according to the changed set of motion vectors. - View Dependent Claims (36)
-
-
40. A method for processing two fields of interlaced video, comprising:
-
computing motion vectors describing motion of image characteristics from a field to another field of opposite sense;
removing from the motion vectors an offset corresponding to one half of a line and having a sign according to an orientation of the y-axis of the image space and which field includes the top line;
using the motion vectors to generate a sampling region at a desired output time;
transforming the sampling region using the motion vectors to a sample time at one of the fields;
transforming the sampling region using the motion vectors to a sample time at the other of the fields;
determining a field sense of an output field to be generated at the desired output time;
translating the transformed sampling region for the field with a field sense opposite the field sense of the output field by an offset of one half of a line and having a sign determined by an orientation of the y-axis of the image space and which field includes the top line; and
generating the output field using the transformed and translated sampling regions and the input fields. - View Dependent Claims (41, 42, 43)
-
-
44. A method for processing two fields of interlaced video, comprising:
-
computing motion vectors describing motion of image characteristics from a field to another field of opposite sense;
removing from the motion vectors an offset corresponding to one half of a line and having a sign according to an orientation of the y-axis of the image space and which field includes the top line;
selecting a time of one of the fields as a desired output time;
transforming a sampling region specified a the selected time using the motion vectors to a sample time at the field that is being warped;
translating the transformed sampling region by an offset of one half of a line and having a sign determined by an orientation of the y-axis of the image space and which field includes the top line; and
generating the output field at the desired output time using the transformed and translated sampling region and the field that is being warped. - View Dependent Claims (45, 46, 47, 48, 49, 50, 51, 52)
-
Specification