Method and assembly for video encoding, the video encoding including texture analysis and texture synthesis, and corresponding computer program and corresponding computer-readable storage medium

• US 7,949,053 B2
• Filed: 08/19/2005
• Issued: 05/24/2011
• Est. Priority Date: 02/28/2003
• Status: Expired due to Fees

• Current Assignee: Fraunhofer Gesellschaft Zur Foerderung Der Angewandten Forsching E.V.
• Original Assignee: Fraunhofer Gesellschaft Zur Foerderung Der Angewandten Forsching E.V.
• Inventors: Ndjiki-Nya, Patrick, Makai, Bela, Wiegand, Thomas, Smolic, Aljoscha, Schwarz, Heiko
• Primary Examiner(s): VO, TUNG T
• Application Number:
  US11/208,103
• Publication Number:
  US 20060039617A1
• Time in Patent Office: 2,104 Days
• Field of Search: 3752401-24029, 382/232
• US Class Current: 375/240.26
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/27 involving both synthetic and natural picture components, e.g. synthetic natural hybrid coding [SNHC]
• CPC Class Codes: H04N 19/537 Motion estimation other than block-based
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards

Abstract

The invention relates to a method and assembly for video coding comprising a texture analysis, texture synthesis, corresponding computer program and a computer-readable recording medium. Said invention can be used, in particular for reducing a data rate during a video data transmission. An analysis of video scene texture by an encoder is performed in such a way that synthesized texture areas are determined. The video scene coding is carried out with the aid of data for said synthesized texture areas. Information on the texture of said areas, metadata used for describing the determined areas and the synthesized texture are generated. Coded data and metadata are evaluated by a decoder in such a way that the video scenes are reconstructed by synthetically generated textures by means of evaluation of metadata for determined areas.

Claims

1. A method for video encoding, comprising:

performing a texture analysis of video scenes to identify areas of synthesizable textures;

encoding the video scenes and generating meta data for describing the areas identified and for describing the synthesizable textures using information on identified areas of synthesizable textures, and information on the textures of these areas; and

ensuring temporal consistency of recognizing synthesizable textures in a sequence of frames by means of a texture catalogue, by

storing the synthesizable textures of the identified areas of synthesizable textures in a first frame of the sequence in the texture catalogue in order to initialize same;

comparing the synthesizable textures of the identified areas of synthesizable textures in the following frames of the sequence with the synthesizable textures stored in the texture catalogue;

in the event of a match, assigning the respective synthesizable texture of an identified area of synthesizable texture among the following frames of the sequence to the respective synthesizable texture stored in the texture catalogue; and

in the event of no match, storing the respective synthesizable texture of an identified area of synthesizable texture among the following frames of the sequence in the texture catalogue.

2. The method as claimed in claim 1, wherein performing the texture analysis comprises subdividing frames into blocks using a multi-resolution quadtree.

3. The method as claimed in claim 2, wherein the subdividing of a block is broken off if

the blocks resulting from the next subdivision step exhibit similar color and/or texture properties as this block, or

the sizes of blocks of the next subdivision step fall short of a predefined value.

4. The method as claimed in claim 3, wherein blocks, the subdividing of which has been broken off because of similar color and/or texture properties, are marked as homogeneous blocks, and the blocks which differ from the homogeneous blocks once the subdividing of the frame has been completed, are marked as non-classifiable blocks.

5. The method as claimed in claim 3, wherein after the subdivision of a frame, a similarity assessment is performed for those blocks whose subdivision was broken off because of similar color and/or texture properties, and wherein similar blocks are combined into synthesizable areas.

6. The method as claimed in claim 5, wherein the similarity of blocks is assessed on the basis of MPEG-7 descriptors.

7. The method as claimed in claim 6, wherein the similarity of blocks is assessed on the basis of the MPEG-7 descriptors “

Edge Histogram”

texture descriptor (EH) and/or “

SCalable Color”

descriptor (SCC).

8. The method as claimed in claim 1, wherein the identified areas of synthesizable textures of a frame of a video sequence are matched with areas of synthesizable textures of preceding frames of the video sequence.

9. The method as claimed in claim 1, wherein the first and last frames (the so-called key frames) of a “

Group of Frames”

(GoF) are encoded on the basis of a mean squared error method (MSE based), and intervening B frames comprising identified synthesizable areas are partially synthesized.

10. The method as claimed in claim 1, wherein synthesizable areas in the partially synthesized frames are adapted to corresponding texture areas in the key frames by suitable warping.

11. The method as claimed in claim 10, wherein the warping is performed by means of a planar perspective motion model described by the following equations:

x′

=[(a₁+a₃x+a₄y)/(1+a₇x+e₈y)]+x,

y′

=[(a₂+a₅x+a₆y)/(1+a₇x+e₈y)]+y,

wherein

(x, y) represent the coordinates of the starting point,

(x′

, y′

) represent the transformed coordinates of the starting point, and

a₁, . . . , a₈ represent model parameters.

12. The method as claimed in claim 10, wherein the warping causes the texture of the first or last frames of the current GoF to be warped in the direction of the synthesizable texture area identified, wherein each such texture area is associated a motion parameter set and a control parameter, the control parameter indicating whether the first or last frame of a GoF is used for texture synthesis.

13. The method as claimed in claim 12, wherein for texture synthesis, that frame of a GoF is used for which a smaller differential signal is determined between the synthesized and the original texture regions.

14. The method as claimed in claim 1, wherein the following are generated per texture area as meta data of areas with synthesizable textures:

a segmentation mask,

a motion parameter set, and/or

a control parameter.

15. The method as claimed in claim 1, further comprising, at the decoder side, the steps of assessing the encoded data and meta data, and reconstructing the video scenes by synthetically generating textures for areas identified by assessing the meta data.

17. A computer program enabling a computer, once it has been loaded into the computer'"'"'s memory, to perform a method as claimed in claim 1.

18. A computer-readable storage medium having a program stored thereon which enables the computer, once it has been loaded into the computer'"'"'s memory, to perform a method as claimed in claim 1.

16. An apparatus for video encoding, comprising:

an analyzer for performing a texture analysis of video scenes to identify areas of synthesizable textures;

an encoder for encoding the video scenes and a generator for generating meta data for describing the areas identified and for describing the synthesizable textures using information on identified areas of synthesizable textures, and information on the textures of these areas; and

a unit for ensuring temporal consistency of recognizing synthesizable textures in a sequence of frames by means of a texture catalogue, by

storing the synthesizable textures of the identified areas of synthesizable textures in a first frame of the sequence in the texture catalogue in order to initialize same;

comparing the synthesizable textures of the identified areas of synthesizable textures in the following frames of the sequence with the synthesizable textures stored in the texture catalogue;

in the event of a match, assigning the respective synthesizable texture of an identified area of synthesizable texture among the following frames of the sequence to the respective synthesizable texture stored in the texture catalogue; and

in the event of no match, storing the respective synthesizable texture of an identified area of synthesizable texture among the following frames of the sequence in the texture catalogue.

19. A method for video encoding, comprising:

performing a texture analysis of video scenes to identify areas of synthesizable textures;

encoding the video scenes and generating meta data for describing the areas identified and for describing the synthesizable textures using information on identified areas of synthesizable textures, and information on the textures of these areas,

the step of generating the meta data comprising the step of estimating motion parameters describing a warping so as to adapt synthesizable areas in frames of a Group of Frames to corresponding texture areas in first or last frames of this group by means of the warping, the motion parameters being part of the meta data.

20. The method as claimed in claim 19, wherein performing the texture analysis comprises subdividing frames into blocks using a multi-resolution quadtree.

21. The method as claimed in claim 20, wherein the subdividing of a block is broken off if

the blocks resulting from the next subdivision step exhibit similar color and/or texture properties as this block, or

the sizes of blocks of the next subdivision step fall short of a predefined value.

22. The method as claimed in claim 21, wherein blocks, the subdividing of which has been broken off because of similar color and/or texture properties, are marked as homogeneous blocks, and the blocks which differ from the homogeneous blocks once the subdividing of the frame has been completed, are marked as non-classifiable blocks.

23. The method as claimed in claim 21, wherein after the subdivision of a frame, a similarity assessment is performed for those blocks whose subdivision was broken off because of similar color and/or texture properties, and wherein similar blocks are combined into synthesizable areas.

24. The method as claimed in claim 23, wherein the similarity of blocks is assessed on the basis of MPEG-7 descriptors.

25. The method as claimed in claim 24, wherein the similarity of blocks is assessed on the basis of the MPEG-7 descriptors “

Edge Histogram”

texture descriptor (EH) and/or “

SCalable Color”

descriptor (SCC).

26. The method as claimed in claim 19, wherein the identified areas of synthesizable textures of a frame of a video sequence are matched with areas of synthesizable textures of preceding frames of the video sequence.

27. The method as claimed in claim 19, wherein the first and last frames (the so-called key frames) of a “

Group of Frames”

(GoF) are encoded on the basis of a mean squared error method (MSE based), and intervening B frames comprising identified synthesizable areas are partially synthesized.

28. The method as claimed in claim 27, wherein the temporal consistency of recognizing synthesizable textures of a Group of Frames (GoF) is ensured by means of a texture catalogue.

29. The method as claimed in claim 19, wherein the warping is performed by means of a planar perspective motion model described by the following equations:

x′

=[(a₁+a₃x+a₄y)/(1+a₇x+a₈y)]+x,

y′

=[(a₂+a₅x+a₆y)/(1+a₇x+a₈y)]+y,

wherein

(x, y) represent the coordinates of the starting point,

(x′

, y′

) represent the transformed coordinates of the starting point, and

a₁, . . . , a₈ represent model parameters.

30. The method as claimed in claim 29, wherein the warping causes the texture of the first or last frames of the current GoF to be warped in the direction of the synthesizable texture area identified, wherein each such texture area is associated a motion parameter set and a control parameter, the control parameter indicating whether the first or last frame of a GoF is used for texture synthesis.

31. The method as claimed in claim 30, wherein for texture synthesis, that frame of a GoF is used for which a smaller differential signal is determined between the synthesized and the original texture regions.

32. The method as claimed in claim 19, wherein the following are generated per texture area as meta data of areas with synthesizable textures:

a segmentation mask,

a motion parameter set, and/or

a control parameter.

36. A computer program enabling a computer, once it has been loaded into the computer'"'"'s memory, to perform a method as claimed in claim 19.

38. A computer-readable storage medium having a program stored thereon which enables the computer, once it has been loaded into the computer'"'"'s memory, to perform a method as claimed in claim 19.

33. A method for video decoding, comprising:

assessing encoded data of video scenes and meta data for describing identified areas of synthesizable textures in the video scenes and for describing the synthesizable textures of these areas; and

reconstructing the video scenes by synthetically generating synthetically generated textures for the areas identified,

wherein the meta data comprise motion parameters describing a warping so as to adapt synthesizable areas in frames of a Group of Frames to corresponding texture areas in first or last frames of this group by means of the warping, and

wherein the step of reconstructing comprises the step of warping the corresponding texture areas in the first or last frames of the group in the direction of the adapted synthesizable areas in the frames of the Group of Frames using the motion parameters.

37. A computer program enabling a computer, once it has been loaded into the computer'"'"'s memory, to perform a method as claimed in claim 33.

39. A computer-readable storage medium having a program stored thereon which enables the computer, once it has been loaded into the computer'"'"'s memory, to perform a method as claimed in claim 33.

40. The method as claimed in claim 33, wherein the first and last frames (the so-called key frames) of a “

Group of Frames”

(GoF) are encoded into the encoded data on the basis of a mean squared error method (MSE based), and intervening frames between the first and last frames of the GoF comprise B frames comprising the identified areas of synthesizable textures, and the method comprises decoding the frames of the GoF.

41. The method as claimed in claim 33, wherein the warping is performed by means of a planar perspective motion model described by the following equations:

x′

=[(a₁+a₃x+a₄y)/(1+a₇x+a₈y)]+x,

y′

=[(a₂+a₅x+a₆y)/(1+a₇x+a₈y)]+y,

wherein

(x, y) represent the coordinates of the starting point,

(x′

, y′

) represent the transformed coordinates of the starting point, and

a₁, . . . , a₈ represent model parameters comprised by the motion parameters.

42. The method as claimed in claim 33, wherein the warping is performed groups-of-frames-wise by causing the texture of first or last frames of a current GoF to be warped into the direction of the identified areas of synthesizable texture in a frame of the current GoF between the first and last frames, wherein the reconstruction comprises checking a control parameter within the meta data to decide as to whether the first or last frame of the current GoF is to be used for texture synthesis.

34. An assembly for video encoding, comprising:

an analyzer for performing a texture analysis of video scenes to identify areas of synthesizable textures;

an encoder for encoding the video scenes and a generator for generating meta data for describing the areas identified and for describing the synthesizable textures using information on identified areas of synthesizable textures, and information on the textures of these areas; and

the generator for generating the meta data being configured to estimate motion parameters describing a warping so as to adapt synthesizable areas in frames of a Group of Frames to corresponding texture areas in first or last frames of this group by means of the warping, the motion parameters being part of the meta data.

35. An assembly for video decoding, comprising:

an assessor for assessing encoded data of video scenes and meta data for describing identified areas of synthesizable textures in the video scenes and for describing the synthesizable textures of these areas; and

a reconstructor for reconstructing the video scenes by synthetically generating synthetically generated textures for the areas identified,

wherein the meta data comprise motion parameters describing a warping so as to adapt synthesizable areas in frames of a Group of Frames to corresponding texture areas in first or last frames of this group by means of the warping, and

wherein the reconstructor is configured to warp the corresponding texture areas in the first or last frames of the group in the direction of the adapted synthesizable areas in the frames of the Group of Frames using the motion parameters.

43. The assembly as claimed in claim 35, wherein the first and last frames (the so-called key frames) of a “

Group of Frames”

(GoF) are encoded into the encoded data on the basis of a mean squared error method (MSE based), and intervening frames between the first and last frames of the GoF comprise B frames comprising the identified areas of synthesizable textures, and the assembly comprises a decoder configured to decode the frames of the GoF.

44. The assembly as claimed in claim 35, wherein the reconstructor is configured to perform the warping by means of a planar perspective motion model described by the following equations:

x′

=[(a₁+a₃x+a₄y)/(1+a₇x+a₈y)]+x,

y′

=[(a₂+a₅x+a₆y)/(1+a₇x+a₈y)]+y,

wherein

(x, y) represent the coordinates of the starting point,

(x′

, y′

) represent the transformed coordinates of the starting point, and

a₁, . . . , a₈ represent model parameters comprised by the motion parameters.

45. The assembly as claimed in claim 35, wherein the reconstructor is configured to perform the warping groups-of-frames-wise by causing the texture of first or last frames of a current GoF to be warped into the direction of the identified areas of synthesizable texture in a frame of the current GoF between the first and last frames, wherein the reconstructor is configured to check a control parameter within the meta data to decide as to whether the first or last frame of the current GoF is to be used for texture synthesis.