Fast motion-compensated video frame interpolator

• US 6,618,439 B1
• Filed: 07/06/1999
• Issued: 09/09/2003
• Est. Priority Date: 07/06/1999
• Status: Expired due to Term

• Current Assignee: MStar Semiconductor Incorporated (MediaTek, Inc.)
• Original Assignee: Industrial Technology Research Institute
• Inventors: Lin, Chia-Wen, Kuo, Chung-Chieh, Kuo, Tien-Ying
• Primary Examiner(s): Philippe, Gims S.
• Application Number:
  US09/348,630
• Time in Patent Office: 1,526 Days
• Field of Search: H04/N.712, H04/B.166, 375/240.16, 375/240.15, 375/240.29, 375/240.26, 375/240.27, 375/240.09, 375/240.12, 375/240.24, 348/402.1, 348/407.1, 348/414.1, 348/415.1, 348/416.1, 348/420.1, 348/417.1, 348/425.1, 348/700, 348/422.1, 382/236, 382/268, 382/242, 382/243
• US Class Current: 375/240.16
• CPC Class Codes: H04N 19/537 Motion estimation other than block-based
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures

Abstract

A block-based motion-compensated frame interpolation method and apparatus using a block-based video coder operating in low bit rates. Smooth movement of objects between video frames can be obtained without the complexity of pixel-wise interpolation motion estimation that is present in standard motion-compensated frame interpolation (MCI). An additional motion search for interpolating all of the individual pixel trajectories is not required because the interpolation uses block-based motion vector information from a standard codec such as H.26x/MPEG. Video quality is improved by increasing smoothness and the frame rate is increased without a substantial increase in the computational complexity. The proposed block-based MCI method maps from block-wise motion to pixel-wise motion in a motion vector mapping unit. A morphological closure operation and pattern block refinement segmentation of the blocks are provided to close holes in the moving object block and replace the morphologically closed motion block with the most similar pattern selected from a group of 34 patterns. Experimental results show that the visual quality of coded low-bit-rate video can be significantly improved as compared to the frame repetition scheme at the expense of a small increase in the complexity of the decoder.

Claims

1. A method of block-based motion-compensated interpolation of a video signal based on blockwise motion vectors and frame information of a plurality of frames being provided by a block-based video decoder, said method comprising:

(a) performing a segmentation operation on said plurality of frames of said video signal to identify an initial moving object block and background information blocks, wherein said background information blocks are identified as a stationary block (SB), an uncovered block (UB), and a covered block (CB),

(b) mapping a motion vector of one of said blockwise motion vectors to provide an output of a mapped moving object block (MO) whose pixels each have said motion vector mapped thereto,

(c) classifying said mapped moving object block (MO) obtained in step (b) and said background information blocks obtained in step (a) to identify an interpolated mapped moving object block (Moti) and interpolated background information blocks including an interpolated stationary block (SBti), an interpolated uncovered block (UBti), and an interpolated covered block (CBti), and

(d) processing Moti, SBti, UBti, CBti and said frame information from said plurality of frames to generate an interpolated frame relative to said one of said frames, wherein the segmentation operation further comprises;

(i) performing a morphological closure operation by removing holes in said initial moving object block to obtain a morphologically closed segmented moving object block,

(ii) performing pattern block refinement by comparing said morphologically closed segmented moving object block obtained in step (i) with a plurality of pattern blocks, and obtaining a pattern block having a closest matching pattern to said morphologically closed segmented moving object block, and

(iii) replacing said morphologically closed segmented moving object with said pattern block selected in step (ii).

2. A method according to claim 1, wherein said blockwise motion vectors and said frame information received in step (a) are provided by one of an MPEG and a H.26x video decoder.

3. A method according to claim 1 further comprising providing said mapped moving object block (MO) produced by step (b) with a shape corresponding to a shape of said closest matching pattern block.

4. A method according to claim 1 wherein said plurality of pattern blocks comprises 34 patterns.

5. A method according to claim 1 wherein each pattern block of said plurality of pattern blocks is a macroblock comprising 16×

16 pixels provided in 16 sub-blocks arranged in a 4×

4 matrix, and each one of the sub-blocks comprises 16 pixels arranged in a 4×

4 matrix.

6. A method according to claim 1 wherein step (d) includes:

performing gap closure of said interpolated moving object block (MOti) to obtain increased solid areas to improve a quality of said interpolated frame.

15. A method according to claim 1, wherein said blockwise motion vectors and said frame information received in step (a) are provided by one of MPEG-1, MPEG-2, H.261 and H.263 video decoder.

7. An apparatus for performing block-based motion-compensated frame interpolation of a video signal based on blockwise motion vectors and frame information of a plurality of frames of said video signal, said apparatus comprising:

(a) segmentation means for performing a segmentation operation on said plurality of frames of said video signal to identify an initial moving object block and background information blocks for one of said frames, said background information blocks comprising a stationary block (SB), an uncovered block (UB) and a covered block (CB);

(b) mapping means for mapping a motion vector of one of said blockwise motion vectors to each pixel of said initial moving object block to provide a mapped moving object block whose pixels each have said motion vector mapped thereto;

(c) classification means for processing said mapped moving object block (MO) output from said mapping means and said background information blocks obtained from said segmentation means to identify an interpolated mapped moving object block (MOti) and interpolated background information blocks including an interpolated stationary block (SBti), an interpolated uncovered block (UBti), and an interpolated covered block (CBti);

(d) motion compensated interpolation means for processing MOti, SBti, UBti, CBti and said frame information relating to said plurality of frames to generate an interpolated frame relative to said one of said frames;

(e) a motion vector replacement unit for comparing said one of said blockwise motion vectors with a set of predetermined criteria to determine whether a value of said one of said blockwise motion vectors requires replacement with a corrected value; and

(f) a residue map which maps prediction errors obtained from a block-based video decoder and outputs said mapped prediction errors to said motion vector replacement unit.

8. An apparatus according to claim 7, further comprising:

a morphological closure unit for processing said initial motion moving object block output by said segmentation means to obtain a morphologically closed segmented moving object block.

10. An apparatus according to claim 7, wherein said motion compensated interpolation means includes:

a gap closure unit for processing gaps in said interpolated moving object block (MOti) to obtain increased solid areas in said interpolated moving object block (MOti) to improve a quality of said interpolated frame.

14. An apparatus according to claim 7, wherein said blockwise motion vectors and said frame information are provided by a block-based video decoder comprising one of an MPEG video decoder and a H.26x video decoder.

16. An apparatus according to claim 7, wherein said blockwise motion vectors and said frame information are provided by a block-based video decoder comprising one of MPEG-1, MPEG-2, H.261 and H.263 video decoder.

9. An apparatus for performing block-based motion-compensated frame interpolation of a video signal based on blockwise motion vectors and frame information of a plurality of frames of said video signal, said apparatus comprising:

(a) segmentation means for performing a segmentation operation on said plurality of frames of said video signal to identify an initial moving object block and background information blocks for one of said frames, said background information blocks comprising a stationary block (SB), an uncovered block (UB) and a covered block (CB),

(b) mapping means for mapping a motion vector of one of said blockwise motion vectors to each pixel of said initial moving object block to provide a mapped moving object block whose pixels each have said motion vector mapped thereto,

(c) classification means for processing said mapped moving object block (MO) output from said mapping means and said background information blocks obtained from said segmentation means to identify an interpolated mapped moving.object block (MOti) and interpolated background information blocks including an interpolated stationary block (SBti), an interpolated uncovered block (UBti), and an interpolated covered block (CBti),

(d) motion compensated interpolation means for processing MOti, SBti, UBti, CBti and said frame information relating to said plurality of frames to generate an interpolated frame relative to said one of said frames;

(e) a morphological closure unit for processing said initial motion moving object.block output by said segmentation means to obtain a morphologically closed segmented moving object block; and

(f) a template matching unit for processing said morphologically closed segmented moving object block output by said morphological closure unit, wherein

said template matching unit compares the morphologically closed segmented moving object block with a plurality of pattern blocks to obtain a most similar pattern block of said plurality of pattern blocks; and

said template matching unit outputs said most similar pattern block of said plurality of pattern blocks to said motion vector mapping unit in place of said morphologically closed segmented moving object block.

11. An apparatus according to claim 9, wherein said template matching unit comprises 34 pattern blocks in storage.

12. An apparatus according to claim 9 wherein each pattern block of said plurality of pattern blocks comprises a macroblock of 16×

16 pixels provided in 16 sub-blocks arranged in a 4×

4 matrix; and

each sub-block comprises 16 pixels arranged in a 4×

4 matrix.

13. An apparatus according to claim 9 wherein said mapped moving object block (MO) output by said mapping means has a shape corresponding to a shape of said most similar pattern block output by said segmentation means.