FRAME INTERPOLATION DEVICE AND METHOD

• US 20120008689A1
• Filed: 03/25/2011
• Published: 01/12/2012
• Est. Priority Date: 07/06/2010
• Status: Abandoned Application

• Current Assignee: Mitsubishi Electric Corporation
• Original Assignee: Mitsubishi Electric Corporation
• Inventors: Ono, Yoshiki, Kubo, Toshiaki, Minami, Koji, NASU, Osamu
• Application Number:
  US13/071,851
• Publication Number:
  US 20120008689A1
• Time in Patent Office: Days
• US Class Current: 375/240.16
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/53 Multi-resolution motion estimation; Hierarchical motion estimation
• CPC Class Codes: H04N 19/57 Motion estimation characterised by a search window with variable size or shape
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence

Abstract

To interpolate a frame between two frames of a video signal, sets of reference images are generated, each set having a different resolution level. Motion between the two frames is estimated at each resolution level by using these sets of reference images. For each pixel processed at each resolution level, multiple motion vector candidates are obtained. Information indicating the multiple motion vector candidates is used to select motion search ranges at the next higher resolution level. To determine the motion search range for a pixel, selected motion vector candidates pertaining both to the pixel itself and to its neighboring pixels are used. An interpolated frame of high image quality is thereby obtainable without increased computation and with reduced risk of major image defects due to erroneous motion estimation.

Claims

1. A frame interpolation apparatus for generating an interpolated frame between a first frame and a second frame in a video signal, the second frame temporally preceding the first frame, the frame interpolation apparatus comprising:

a reference image generator for receiving image signals of the first frame and the second frame and generating therefrom a plurality of sets of reference images, the reference images in each one of the sets having mutually identical resolution, the reference images in different ones of the sets having different resolutions;

a motion estimation unit for performing motion estimation based on the plurality of sets of reference images; and

an interpolated frame generator for generating an image signal of the interpolated frame, each pixel on the interpolated frame being based on at least one motion vector candidate obtained as a result of motion estimation performed by the motion estimation unit using the set of reference images of highest resolution;

wherein

the motion estimation unit sequentially generates information representing results of motion estimation by proceeding sequentially from motion estimation using the reference images of lowest resolution to motion estimation using the reference images of the highest resolution; and

in performing motion estimation by using the reference images of each resolution, the motion estimation unit determines a search range, for each pixel processed on the second frame, by

using information indicating a motion vector candidate obtained for the pixel being processed as a result of motion estimation performed using the set of reference images of next lower resolution, and

also using information indicating a motion vector candidate obtained for a pixel neighboring the pixel being processed as a result of motion estimation performed using the set of reference images of the next lower resolution.

2. The frame interpolation apparatus of claim 1, wherein the motion estimation unit, in determining the search range in the motion estimation performed by using the reference images of each resolution, determines, for each pixel processed in the second frame, a search range including a prescribed range centered on a position to which the pixel being processed moved according to a motion vector candidate estimated for the pixel being processed using the set of reference images of the next lower resolution, and a prescribed range centered on a position to which the pixel being processed moved according to a motion vector candidate estimated for a pixel neighboring the pixel being processed using the set of reference images of the next lower resolution.

3. The frame interpolation apparatus of claim 1, wherein the motion estimation unit further comprises a motion vector candidate information selector for selecting a predetermined number of items of information from information indicating motion vector candidates estimated for pixels neighboring the pixel being processed using the set of reference images of the next lower resolution, the selected items of information being used to determine the search range for the pixel being processed.

4. The frame interpolation apparatus of claim 3, wherein the motion vector candidate information selector selects,

from the information indicating motion vector candidates estimated for the pixels neighboring the pixel being processed,

a certain number of items of information including motion vector candidates of greatest similarity that differ most greatly from the motion vector candidate of greatest similarity among the motion vector candidates estimated for the pixel being processed, and

information indicating motion vector candidates estimated for the pixels in point-symmetric positions to the neighboring pixels to which the selected certain number of items of information pertain, the pixel being processed being the center of symmetry.

5. The frame interpolation apparatus of claim 4, wherein the motion estimation unit further comprises a search range limiter for determining the search range according to the information indicating a plurality of the motion vector candidates selected by the motion vector candidate information selector.

6. The frame interpolation apparatus of claim 5, wherein the search range limiter determines, as the search range, a region including prescribed ranges centered on a plurality of positions to which the pixel being processed moved according to motion vector candidates indicated by the information selected by the motion vector candidate information selector.

7. The frame interpolation apparatus of claim 1, wherein the reference image generator generates the plurality of sets of references images by repeatedly reducing the first frame and the second frame by a predetermined reduction ratio, taking mean values of a plurality of pixels as pixel values in the reduced reference images.

8. The frame interpolation apparatus of claim 1, wherein the reference image generator also receives an image signal of a third frame temporally preceding the second frame and generates, in each one of the sets of reference images, a reference image derived from the third frame as well as reference images derived from the first and second frames.

9. The frame interpolation apparatus of claim 1, wherein when the interpolated frame includes a pixel that is not a motion destination of any pixel on the first frame according to any motion vector candidate, the interpolated frame generator obtains a motion vector candidate for the pixel in the interpolated frame by interpolation, using motion vector candidates obtained for pixels neighboring the pixel in the interpolated frame.

10. The frame interpolation apparatus of claim 1, wherein when the motion estimation unit outputs two or more motion vector candidates for a pixel on the interpolated frame, the interpolated frame generator selects, from among pairs of reference pixels determined according to the two or more motion vector candidates, a pair of reference pixels with pixel values that differ least from each other, and determines a pixel value for the pixel on the interpolated frame from the pixel values of the selected pair of reference pixels.

11. A frame interpolation method for generating an interpolated frame between a first frame and a second frame in a video signal, the second frame temporally preceding the first frame, the frame interpolation method comprising:

receiving image signals of the first frame and the second frame and generating therefrom a plurality of sets of reference images, the reference images in each one of the sets having mutually identical resolution, the reference images in different ones of the sets having different resolutions;

performing motion estimation based on the plurality of sets of reference images; and

generating an image signal of the interpolated frame, each pixel on the interpolated frame being based on at least one motion vector candidate obtained as a result of the motion estimation performed using the set of reference images of highest resolution;

wherein

performing the motion estimation further comprises sequentially generating information representing results of motion estimation by proceeding sequentially from motion estimation using the reference images of lowest resolution to motion estimation using the reference images of the highest resolution; and

performing the motion estimation by using the reference images of each resolution further comprises determining a search range, for each pixel processed on the second frame, by

using information indicating a motion vector candidate obtained for the pixel being processed as a result of motion estimation performed using the set of reference images of next lower resolution, and

also using information indicating a motion vector candidate obtained for a pixel neighboring the pixel being processed as a result of motion estimation performed using the set of reference images of the next lower resolution.

12. The frame interpolation method of claim 11, wherein the search range determined for each pixel processed in the second frame in the motion estimation performed by using the reference images of each resolution further includes a prescribed range centered on a position to which the pixel being processed moved according to a motion vector candidate estimated for the pixel being processed using the set of reference images of the next lower resolution, and a prescribed range centered on a position to which the pixel being processed moved according to a motion vector candidate estimated for a pixel neighboring the pixel being processed using the set of reference images of the next lower resolution.

13. The frame interpolation method of claim 11, wherein performing the motion estimation further comprises selecting a predetermined number of items of information from information indicating motion vector candidates estimated for pixels neighboring the pixel being processed using the set of reference images of the next lower resolution, and using the selected items of information.

14. The frame interpolation method of claim 13, wherein the predetermined number of items of information include:

a certain number of items of information including motion vector candidates of greatest similarity that differ most greatly from the motion vector candidate of greatest similarity among the motion vector candidates estimated for the pixel being processed; and

information indicating motion vector candidates estimated for the pixels in point-symmetric positions to the neighboring pixels to which the selected certain number of items of information pertain, the pixel being processed being the center of symmetry.

15. The frame interpolation method of claim 14, wherein the search range is determined according to the selected predetermined number of items of information.

16. The frame interpolation method of claim 15, wherein the search range includes prescribed ranges centered on a plurality of positions to which the pixel being processed moved according to motion vector candidates indicated by the selected predetermined number of items of information.

17. The frame interpolation method of claim 11, wherein receiving the image signals of the first frame and the second frame and generating therefrom the plurality of sets of references images further comprises repeatedly reducing the first frame and the second frame by a predetermined reduction ratio, taking mean values of a plurality of pixels as pixel values in the reduced reference images.

18. The frame interpolation method of claim 11, wherein receiving the image signals of the first frame and the second frame and generating therefrom the plurality of sets of references images further comprises receiving an image signal of a third frame temporally preceding the second frame and generating, in each one of the sets of reference images, a reference image derived from the third frame as well as reference images derived from the first and second frames.

19. The frame interpolation method of claim 11, wherein when the interpolated frame includes a pixel that is not a motion destination of any pixel on the first frame according to any motion vector candidate, generating the image signal of the interpolated frame further comprises obtaining a motion vector candidate for the pixel in the interpolated frame by interpolation, using motion vector candidates obtained for pixels neighboring the pixel in the interpolated frame.

20. The frame interpolation method of claim 11, wherein when two or more motion vector candidates are obtained for a pixel on the interpolated frame, generating an image signal of the interpolated frame includes selecting, from among pairs of reference pixels determined according to the two or more motion vector candidates, a pair of reference pixels with pixel values that differ least from each other, and determining a pixel value for the pixel on the interpolated frame from the pixel values of the selected pair of reference pixels.

21. A computer-readable recording medium storing a program executable to perform frame interpolation by the method of claim 11.