Mesh node coding to enable object based functionalities within a motion compensated transform video coder

• US 6,148,026 A
• Filed: 12/29/1997
• Issued: 11/14/2000
• Est. Priority Date: 01/08/1997
• Status: Expired due to Term

• Current Assignee: AT&T Corporation (AT&T, Inc.), Sharp Laboratories
• Original Assignee: AT&T Corporation (AT&T, Inc.), Sharp Laboratories
• Inventors: van Beek, Petrus J. L., Puri, Atul, Tekalp, A. Murat
• Primary Examiner(s): Tung, Bryan
• Application Number:
  US08/998,855
• Time in Patent Office: 1,051 Days
• Field of Search: 375/240, 348/384, 348/390, 348/397, 348/402, 348/409, 348/415, 348/416, 348/420, 348/699, 382/232, 382/233, 382/236, 382/238, 382/241, 382/243
• US Class Current: 375/240
• CPC Class Codes: H04N 19/29 involving scalability at the object level, e.g. video object layer [VOL]
• CPC Class Codes: H04N 19/537 Motion estimation other than block-based
• CPC Class Codes: H04N 19/54 using feature points or meshes

Abstract

A method and apparatus for coding video data permits coding of video information with optional, enhanced functionalities. Video data is coded as base layer data and enhancement layer data. The base layer data includes convention motion compensated transform encoded texture and motion vector data. Optional enhancement layer data contains mesh node vector data. Mesh node vector data of the enhancement layer may be predicted based on motion vectors of the base layer. Thus, simple decoders may decode the base layer data and obtain a basic representation of the coded video data. However, more powerful decoders may decode both the base layer and enhanced layer to obtain decoded video permitting functionalities. An embodiment of the present invention provides a back channel that permits a decoder to affect how mesh node coding is performed in the encoder. The decoder may command the encoder to reduce or eliminate encoding of mesh node motion vectors. The back channel finds application in single layer systems and two layer systems.

Claims

1. A video coder that codes video data as coded output, the coded output including block based texture data, block based motion data and mesh node based motion data, the coder responsive to a predetermined command received over a back channel by reducing a number of mesh nodes represented in the mesh node based motion data.

2. The video coder of claim 1, wherein, when the command requests the mesh node motion data to be eliminated, the coder ceases to code video data as mesh node based motion data.

3. The video coder of claim 1, wherein the video coder is a single layer coder.

4. The video coder of claim 1, wherein the video coder is a dual layer coder, the block based texture data and block based motion data being includes in coded base layer data and the mesh node based motion data being included in coded enhancement layer data.

5. A video decoder that decodes coded video data received over a channel, the coded video data including block based texture data, block based motion data and mesh node based motion data, the decoder determining whether the mesh node motion data is necessary for video decoding and, when the mesh node motion data is unnecessary, generating a back channel command representing a request to reduce a number of mesh nodes represented by the node based motion data.

6. The video decoder of claim 5, wherein the command requests elimination of the mesh node motion data.

7. The video decoder of claim 5, wherein the video decoder is a single layer decoder.

8. The video decoder of claims 5, wherein the video decoder is a dual layer decoder, the block based texture data and block based motion data being included in coded base layer data and the mesh node based motion data being included in coded enhancement layer data.

9. A video coder that codes input video data, comprising:

a base layer encoder that receives the input video data and outputs coded base layer data representative of the input video data, the coded base layer data including block based texture data and block based motion data,

an enhancement layer encoder that receives the input video data and outputs coded enhanced layer data including mesh node based motion information,

a multiplexer that merges the coded base layer data and coded enhanced layer data into a unitary data stream and outputs the unitary data stream to a channel.

10. The video coder of claim 9, wherein the base layer encoder outputs the block based motion data to the enhanced layer encoder.

11. The video coder of claim 9, wherein the multiplexer routes contents of a back channel contained within the channel to the enhancement layer encoder.

12. The video coder of claim 11, wherein responsive to a predetermined command, the enhancement layer encoder reduces a number of mesh nodes represented by the mesh node motion data.

13. The video coder of claim 11, wherein responsive to a predetermined command, the enhancement layer encoder is disabled.

14. The video coder of claim 9, wherein the input video data constitutes video object planes.

15. The video coder of claim 9, wherein a representation of the input video data may be generated from the coded base layer data.

16. A video decoder that decodes coded video data, the coded video data including block based motion data and block based texture data included in coded base layer data and mesh based motion data included in coded enhancement layer data, the decoder comprising:

a base layer decoder that decodes the coded base layer data into decoded video data and motion data,

an enhancement layer decoder that decodes the enhancement layer data,

a demutliplexer that receives the coded video data from a channel, routes the coded base layer data to the base layer decoder and routes the coded enhancement layer data to the enhancement layer decoder, and

a compositor that receives decoded outputs from the base layer encoder and the enhancement layer decoder and generates a synthesized video output therefrom.

17. The video decoder of claim 16, wherein the base layer decoder outputs the block based motion data to the enhanced layer decoder.

18. The video decoder of claim 16, wherein the compositor generates commands and outputs the commands to the demultiplexer and the demultiplexer places the commands in a back channel in the channel.

19. The video decoder of claim 18, wherein the compositor generates a command representing a request to reduce a number of mesh nodes represented by the mesh node based motion data.

20. The video decoder of claim 18, wherein the compositor generates a command representing a request to eliminate the mesh node based motion data.

21. The video decoder of claim 16, wherein compositor'"'"'s synthesized video output is generated as video object planes.

22. A video decoder that decodes coded video data, the coded video data including block based motion data and block based texture data included in coded base layer data and mesh based motion data included in coded enhancement layer data, the decoder comprising:

a base layer decoder that decodes the coded base layer data into decoded video data and motion data,

a demutliplexer that receives the coded video data from a channel and routes the coded base layer data to the base layer decoder and ignores the coded enhancement layer data, and

a compositor that receives a decoded output from the base layer encoder and generates a synthesized video output therefrom.

23. In a two-layer video coder that encodes input video data as coded video data, the coded video data including a base layer of block based texture data and block based motion data and an enhancement layer of mesh node based motion data, a method of coding the enhancement layer data, comprising:

generating mesh nodes from the input video data,

receiving the block based motion data,

for at least one mesh node;

identifying a block that is spatially related to the mesh node,

predicting a motion vector for the mesh node based upon a motion vector of the spatially related block,

comparing an actual motion vector of the mesh node against the predicted motion vector, and

encoding a difference between the actual motion vector and the predicted motion vector.

24. The method of claim 23, wherein the predicting step includes steps of:

identifying a motion vector of the spatially related block,

identifying motion vectors of blocks neighboring the spatially related block, and

predicting the motion vector of the mesh node as a predetermined function of the spatially co-located block and its neighbors.

25. The method of claim 24, wherein the neighboring blocks include blocks above, below, to the left and to the right of the related block.

26. The method of claim 24, wherein the neighboring blocks including all blocks circumscribing the related block.

27. The method of claim 24, wherein the neighboring blocks are the blocks circumscribing the related block and are closest to the mesh node'"'"'s position within the related block.

28. The method of claim 24, wherein the predetermined function is an averaging function.

29. The method of claim 24, wherein the predetermined function is a weighted averaging function.

30. The method of claim 24, wherein the predetermined function is a median function.

31. The method of claim 23, wherein a block is an 8 pixel by 8 pixel rectangular area.

32. The method of claim 23, wherein a block is a macroblock, a 16 pixel by 16 pixel rectangular area.

33. In a two-layer video decoder that decodes coded video data, the coded video data including a base layer of block based texture data and block based motion data and an enhancement layer of residual mesh node based motion data, a method of decoding the mesh node based motion data, comprising:

receiving the coded enhancement layer data,

for at least one mesh node;

for at least one mesh node;

identifying a block that is spatially related to the mesh node,

predicting a motion vector for the mesh node based upon a motion vector of the spatially related block, and

adding residual motion data of the node to the predicted motion vector.

34. The method of claim 33, wherein the predicting step includes steps of:

identifying a motion vector of the spatially related block,

identifying motion vectors of blocks neighboring the spatially related block, and

predicting the motion vector of the mesh node as a predetermined function of the spatially co-located block and its neighbors.

35. The method of claim 34, wherein the neighboring blocks include blocks above, below, to the left and to the right of the related block.

36. The method of claim 34, wherein the neighboring blocks including all blocks circumscribing the related block.

37. The method of claim 34, wherein the neighboring blocks are the blocks circumscribing the related block and are closest to the mesh node'"'"'s position within the related block.

38. The method of claim 34, wherein the predetermined function is an averaging function.

39. The method of claim 34, wherein the predetermined function is a weighted averaging function.

40. The method of claim 34, wherein the predetermined function is a median finction.

41. The method of claim 33, wherein a block is an 8 pixel by 8 pixel rectangular area.

42. The method of claim 33, wherein a block is a macroblock, a 16 pixel by 16 pixel rectangular area.