Method and apparatus for update step in video coding using motion compensated temporal filtering

• US 20070053441A1
• Filed: 06/29/2006
• Published: 03/08/2007
• Est. Priority Date: 06/29/2005
• Status: Abandoned Application

• Current Assignee: Nokia Corporation
• Original Assignee: Nokia Corporation
• Inventors: Karczewicz, Marta, Wang, Xianglin, Bao, Yiliang, Ridge, Justin
• Application Number:
  US11/479,126
• Publication Number:
  US 20070053441A1
• Time in Patent Office: Days
• US Class Current: 375/240.240
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/119 Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
• CPC Class Codes: H04N 19/13 Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/521 for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
• CPC Class Codes: H04N 19/523 with sub-pixel accuracy
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/615 using motion compensated temporal filtering [MCTF]
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/82 involving filtering within a prediction loop

Abstract

The present invention provides a method and module for performing the update operation in motion compensated temporal filtering for video coding. The update operation is performed according to coding blocks in the prediction residue frame. Depending on macroblock mode in the prediction step, a coding block can have different sizes. Macroblock modes are used to specify how a macroblock is segmented into blocks. In the prediction step, the reverse direction of the motion vectors is used directly as an update motion vector and therefore no motion vector derivation process is performed. Motion vectors that significantly deviate from their neighboring motion vectors are considered not reliable and excluded from the update step. An adaptive filter is used in interpolating the prediction residue block for the update operation. The adaptive filter is an adaptive combination of a short filter and a long filter.

Claims

1. A method of encoding a digital video sequence using motion compensated temporal filtering for providing a bitstream having video data representative of encoded video sequence, the digital video sequence comprising a plurality of frames, wherein each frame comprises an array of pixels which can be divided into a plurality of macroblocks, said method comprising:

for a macroblock,

selecting a macroblock mode;

segmenting the macroblock into a number of blocks based on the macroblock mode;

performing a prediction operation on said blocks, based on motion compensated prediction with respect to a reference video frame and motion vectors, for providing corresponding blocks of prediction residues; and

updating said video reference frame based on motion compensated prediction with respect to said blocks of prediction residues and the macroblock mode, and further based on a reverse direction of said motion vectors.

2. The method of claim 1, wherein each of the blocks is associated with one of the motion vectors, said method further comprising:

comparing the motion vector associated with one of the blocks with the motion vectors associated with adjacent blocks for providing a differential vector of said one block; and

skipping said updating with respect to said one block if the differential vector is greater than a predetermined value.

3. The method of claim 1, wherein the blocks of prediction residue form a prediction residue frame, said updating comprising:

interpolating sub-pixel locations of said blocks of prediction residues in the prediction residue frame based on an interpolation filter.

4. The method of claim 3, wherein the interpolation filter is adaptively selected from a plurality of filters comprising at least a shorter filter and a longer filter.

5. The method of claim 4, wherein said selection is at least partially based on an energy level of prediction residue in said block.

6. The method of claim 1, further comprising:

limiting amplitude of the prediction residue of a block in said updating to a threshold determined at least based on an energy level of the prediction residue in said block.

7. The method of claim 1, further comprising:

limiting amplitude of the prediction residue of a block in said updating to a threshold determined at least based on a block matching factor of said block.

8. A method of decoding a digital video sequence from video data in a bitstream representative of an encoded video sequence, the encoded video sequence comprising a number of frames, each frame comprising an array of pixels, wherein the pixels in each frame can be divided into a plurality of macroblocks, said method comprising:

for a macroblock,

obtaining a macroblock mode;

segmenting the macroblock into a number of blocks based on the macroblock mode;

decoding motion vectors and prediction residues of the blocks;

performing an update operation on a reference video frame of said blocks, based on motion compensated prediction with respect to the prediction residues of said blocks based on said macroblock mode and a reverse direction of the motion vectors; and

performing a prediction operation on said blocks based on motion compensated prediction with respect to updated reference video frame and the motion vectors.

9. The method of claim 8, wherein each of the blocks is associated with one of the motion vectors, said method further comprising:

comparing the motion vector associated with one of the blocks with the motion vectors associated with adjacent blocks for providing a differential vector of said one block; and

skipping said updating with respect to the said one block if the differential vector is greater than a predetermined value.

10. The method of claim 8, wherein the blocks of prediction residues form a prediction residue frame, said updating comprising:

interpolating sub-pixel locations of said blocks of prediction residues in the prediction residue frame based on an interpolation filter.

11. The method of claim 10, wherein the interpolation filter is adaptively selected from a plurality of filters comprising at least a shorter filter and a longer filter.

12. The method of claim 11, wherein said selection is at least partially based on an energy level of prediction residue in said block.

13. The method of claim 8, further comprising:

limiting amplitude of the prediction residue of a block in said updating to a threshold determined at least based on an energy level of the prediction residue in said block.

14. The method of claim 8, further comprising:

limiting amplitude of the prediction residue of a block in said updating to a threshold determined at least based on a block matching factor of said block.

15. An encoding module for use in encoding a digital video sequence using motion compensated temporal filtering for providing a bitstream having video data representative of encoded video sequence, the digital video sequence comprising a plurality of frames, wherein each frame comprises an array of pixels which can be divided into a plurality of macroblocks, said encoding module comprising:

a mode decision module configured for selecting, for a macroblock, a macroblock mode so as to segment the macroblock into a number of blocks based on the macroblock mode;

a prediction module for performing a prediction operation on said blocks, based on motion compensated prediction with respect to a reference video frame and motion vectors, for providing corresponding blocks of prediction residues; and

an updating module for updating said video reference frame based on motion compensated prediction with respect to said blocks of prediction residues and the macroblock mode, and further based on a reverse direction of said motion vectors.

16. The encoding module of claim 15, wherein each of the blocks is associated with one of the motion vectors, said encoding module further comprising:

a processor for comparing the motion vector associated with one of the blocks with the motion vectors associated with adjacent blocks for providing a differential vector of said one block;

such that when the differential vector is greater than a predetermined value, the updating module is configured to skip said updating with respect to said one block if the differential vector is greater than a predetermined value.

17. The encoding module of claim 15, wherein the blocks of prediction residue form a prediction residue frame, said encoding module further comprising:

an interpolation filter module for interpolating sub-pixel locations of said blocks of prediction residues in the prediction residue frame based on an interpolation filter.

18. The encoding module of claim 17, wherein the interpolation filter is adaptively selected from a plurality of filters comprising at least a shorter filter and a longer filter.

19. The encoding module of claim 18, wherein said selection is at least partially based on an energy level of prediction residue in said block.

20. The encoding module of claim 15, further comprising:

an amplitude control module for limiting amplitude of the prediction residue of a block in said updating to a threshold determined at least based on an energy level of the prediction residue in said block.

21. The encoding module of claim 15, further comprising:

an amplitude control module for limiting amplitude of the prediction residue of a block in said updating to a threshold determined at least based on a block matching factor of said block.

22. A decoding module for use in decoding a digital video sequence from video data in a bitstream representative of an encoded video sequence, the encoded video sequence comprising a number of frames, each frame comprising an array of pixels, wherein the pixels in each frame can be divided into a plurality of macroblocks, said decoding module comprising:

a first decoding sub-module, responsive to the video data, for decoding a macroblock mode so as to segment the macroblock into a number of blocks based on the macroblock mode;

a second decoding sub-module for decoding motion vectors and prediction residues of the blocks;

an updating module for performing an update operation on a reference video frame of said blocks, based on motion compensated prediction with respect to the prediction residues of said blocks based on said macroblock mode and a reverse direction of the motion vectors; and

a prediction module for performing a prediction operation on said blocks based on motion compensated prediction with respect to updated reference video frame and the motion vectors.

23. The decoding module of claim 22, wherein each of the blocks is associated with one of the motion vectors, said decoding module further comprising:

a processor for comparing the motion vector associated with one of the blocks with the motion vectors associated with adjacent blocks for providing a differential vector of said one block;

such that when the differential vector is greater than a predetermined value, the updating module is configured to skip said updating with respect to the said one block.

24. The decoding module of claim 22, wherein the blocks of prediction residues form a prediction residue frame, said decoding module further comprising:

an interpolation filter module for interpolating sub-pixel locations of said blocks of prediction residues in the prediction residue frame based on an interpolation filter.

25. The decoding module of claim 24, wherein the interpolation filter is adaptively selected from a plurality of filters comprising at least a shorter filter and a longer filter.

26. The decoding module of claim 25, wherein said selection is at least partially based on an energy level of prediction residue in said block.

27. The decoding module of claim 22, further comprising:

an amplitude control module for limiting amplitude of the prediction residue of a block in said updating to a threshold determined at least based on an energy level of the prediction residue in said block.

28. The decoding module of claim 22, further comprising:

an amplitude control module for limiting amplitude of the prediction residue of a block in said updating to a threshold determined at least based on a block matching factor of said block.

29. A software application product, comprising a storage medium having a software application for encoding a digital video sequence using motion compensated temporal filtering for providing a bitstream having video data representative of encoded video sequence, the digital video sequence comprising a plurality of frames, wherein each frame comprises an array of pixels which can be divided into a plurality of macroblocks, said software application comprising:

program code for selecting a macroblock mode for a macroblock;

program code for segmenting the macroblock into a number of blocks based on the macroblock mode;

program code for performing a prediction operation on said blocks, based on motion compensated prediction with respect to a reference video frame and motion vectors, for providing corresponding blocks of prediction residues; and

program code for updating said video reference frame based on motion compensated prediction with respect to said blocks of prediction residues and the macroblock mode, and further based on a reverse direction of said motion vectors.

30. The software application product of claim 29, wherein each of the blocks is associated with one of the motion vectors, said software application further comprising:

program code for comparing the motion vector associated with one of the blocks with the motion vectors associated with adjacent blocks for providing a differential vector of said one block and, if the differential vector is greater than a predetermined value, skipping said updating with respect to said one block.

31. A software application product, comprising a storage medium having a software application for decoding a digital video sequence from video data in a bitstream representative of an encoded video sequence, the encoded video sequence comprising a number of frames, each frame comprising an array of pixels, wherein the pixels in each frame can be divided into a plurality of macroblocks, said software application comprising:

program code for obtaining a macroblock mode for a macroblock from the video data;

program code for segmenting the macroblock into a number of blocks based on the macroblock mode;

program code for decoding motion vectors and prediction residues of the blocks;

program code for performing an update operation on a reference video frame of said blocks, based on motion compensated prediction with respect to the prediction residues of said blocks based on said macroblock mode and a reverse direction of the motion vectors; and

program code for performing a prediction operation on said blocks based on motion compensated prediction with respect to updated reference video frame and the motion vectors.

32. The software application product of claim 31, wherein each of the blocks is associated with one of the motion vectors, said software application further comprising:

program code for comparing the motion vector associated with one of the blocks with the motion vectors associated with adjacent blocks for providing a differential vector of said one block and, if the differential vector is greater than a predetermined value, skipping said updating with respect to the said one block

33. An electronic device configured to acquire a digital video sequence, comprising:

an encoding module for encoding the digital video sequence using motion compensated temporal filtering for providing a bitstream having video data representative of encoded video sequence, the digital video sequence comprising a plurality of frames, wherein each frame comprises an array of pixels which can be divided into a plurality of macroblocks, said encoding module comprising;

a mode decision module configured for selecting, for a macroblock, a macroblock mode so as to segment the macroblock into a number of blocks based on the macroblock mode;

a prediction module for performing a prediction operation on said blocks, based on motion compensated prediction with respect to a reference video frame and motion vectors, for providing corresponding blocks of prediction residues; and

an updating module for updating said video reference frame based on motion compensated prediction with respect to said blocks of prediction residues and the macroblock mode, and further based on a reverse direction of said motion vectors.

34. The electronic device of claim 33, further configured to receive video data representation of an encoded video sequence, the mobile terminal further comprising:

a decoding module for decoding the encoded video sequence from video data, the encoded video sequence comprising a number of frames, each frame comprising an array of pixels, wherein the pixels in each frame can be divided into a plurality of macroblocks, said decoding module comprising;

a first decoding sub-module, responsive to the video data, for decoding a macroblock mode so as to segment the macroblock into a number of blocks based on the macroblock mode;

a second decoding sub-module for decoding motion vectors and prediction residues of the blocks;

an updating module for performing an update operation on a reference video frame of said blocks, based on motion compensated prediction with respect to the prediction residues of said blocks based on said macroblock mode and a reverse direction of the motion vectors; and

a prediction module for performing a prediction operation on said blocks based on motion compensated prediction with respect to updated reference video frame and the motion vectors.

35. An encoding module for use in encoding a digital video sequence using motion compensated temporal filtering for providing a bitstream having video data representative of encoded video sequence, the digital video sequence comprising a plurality of frames, wherein each frame comprises an array of pixels which can be divided into a plurality of macroblocks, said encoding module comprising:

means for selecting, for a macroblock, a macroblock mode so as to segment the macroblock into a number of blocks based on the macroblock mode;

means for performing a prediction operation on said blocks, based on motion compensated prediction with respect to a reference video frame and motion vectors, for providing corresponding blocks of prediction residues; and

means for updating said video reference frame based on motion compensated prediction with respect to said blocks of prediction residues and the macroblock mode, and further based on a reverse direction of said motion vectors.

36. The encoding module of claim 35, wherein each of the blocks is associated with one of the motion vectors, said encoding module further comprising:

means for comparing the motion vector associated with one of the blocks with the motion vectors associated with adjacent blocks for providing a differential vector of said one block;

such that when the differential vector is greater than a predetermined value, the updating module is configured to skip said updating with respect to said one block if the differential vector is greater than a predetermined value.

37. A decoding module for use in decoding a digital video sequence from video data in a bitstream representative of an encoded video sequence, the encoded video sequence comprising a number of frames, each frame comprising an array of pixels, wherein the pixels in each frame can be divided into a plurality of macroblocks, said decoding module comprising:

means, responsive to the video data, for decoding a macroblock mode so as to segment the macroblock into a number of blocks based on the macroblock mode;

means for decoding motion vectors and prediction residues of the blocks;

means for performing an update operation on a reference video frame of said blocks, based on motion compensated prediction with respect to the prediction residues of said blocks based on said macroblock mode and a reverse direction of the motion vectors; and

means for performing a prediction operation on said blocks based on motion compensated prediction with respect to updated reference video frame and the motion vectors.

38. The decoding module of claim 37, wherein each of the blocks is associated with one of the motion vectors, said decoding module further comprising:

means for comparing the motion vector associated with one of the blocks with the motion vectors associated with adjacent blocks for providing a differential vector of said one block;

such that when the differential vector is greater than a predetermined value, the updating module is configured to skip said updating with respect to the said one block.