VIDEO CODING SYSTEM USING TEXTURE ANALYSIS AND SYNTHESIS IN A SCALABLE CODING FRAMEWORK

• US 20090180552A1
• Filed: 09/22/2008
• Published: 07/16/2009
• Est. Priority Date: 01/16/2008
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Inventors: Tabatabai, Ali, Visharam, Mohammed Z.
• Granted Patent: US 8,155,184 B2
• Time in Patent Office: Days
• US Class Current: 375/240.260
• CPC Class Codes: H04N 19/12 Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/192 the adaptation method, adaptation tool or adaptation type being iterative or recursive
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

An apparatus or method of efficiently coding high resolution video using texture analysis and synthesis techniques in a scalable video coding framework. A high-resolution video signal is spatially downsampled and encoded into a base-layer. Texture and structure information are extracted from the downsampled signal and base-layer for use by a texture synthesizer. The texture synthesizer is structurally and texturally aware utilizing edge information from a base-layer of the synthesizer to improve synthesis. After synthesis, a video quality assessor directs enhancement-layer coding of unacceptably synthesized areas by alternate (non-textural) coding means, such as conventional AVC or MPEG-2 coding. In one mode, the quality assessor iteratively improves synthesis of certain blocks to make them acceptable for enhancement-layer coding. The apparatus or method outputs a bit stream, or bit streams, containing both the coded base-layer and the enhancement-layer.

Claims

1. A method of efficiently coding a high-resolution video sequence, comprising:

spatial downsampling of a high-resolution video sequence received as input to obtain downsampled video having a lower spatial resolution video signal;

coding of a base-layer of said downsampled video utilizing conventional video coding;

performing texture analysis and classification from said downsampled video;

determining structural information from said decoded base-layer;

performing texture synthesis on blocks, which are determined to be texture synthesis compatible, in response to said structural information;

coding an enhancement-layer in response to structure analysis of the base-layer for acceptably synthesized blocks and using a traditional scalable coding framework for unacceptably synthesized blocks; and

outputting at least one bit stream containing the coded base-layer and enhancement-layer.

2. A method as recited in claim 1, wherein said texture analysis is performed from said downsampled video.

3. A method as recited in claim 1, wherein the determination of texture synthesis compatible blocks comprises performing texture analysis &

classification of blocks into texture blocks which are amenable to texture synthesis (T blocks), and non-texture blocks (NT blocks) on which alternate non-textural coding is to be performed.

4. A method as recited in claim 1, further comprising assessing visual acceptability of texture synthesis and classifying texture blocks as acceptable or non-acceptable.

5. A method as recited in claim 4, further comprising determining non-acceptable blocks which are marginally non-acceptable and performing additional synthesis processing on these blocks to bring these blocks to an acceptable level of visual quality.

6. A method as recited in claim 5, wherein the block is considered unacceptable if it cannot be brought to an acceptable quality level within a desired limit of time or number of iterations.

7. A method as recited in claim 1, wherein said synthesis based on said structural information obtained from said downsampled video preserves the edge information of synthesized regions in response to said structural awareness data in the high resolution video.

8. A method as recited in claim 1, wherein said synthesis utilizes low resolution texture information from the decoded base layer during the synthesis process of the high resolution layer.

9. A method as recited in claim 1, wherein said method is performed in response to execution of programming on a computer configured for processing high-definition video signals.

10. A method of efficiently coding a high-resolution video sequence, comprising:

spatial downsampling of a high-resolution video sequence received as input to obtain downsampled video having a lower spatial resolution video signal;

coding of a base-layer of said downsampled video utilizing conventional video coding;

performing texture analysis of said downsampled video into texture blocks (T blocks) upon which texture synthesis can be performed, and non-texture blocks (NT blocks) upon which alternate non-textural coding is to be performed;

performing texture classification of T blocks into various categories varying from highly rigid to non-rigid so that effective synthesis techniques can be performed dependent upon the classification;

obtaining structural information from said base-layer;

performing texture synthesis in response to said structural information;

performing texture quality assessment of texture blocks to classify visual acceptability of synthesized areas as acceptable, non-acceptable and marginally non-acceptable;

wherein determining of acceptably synthesized blocks is performed by quality assessor programming which utilizes iterative processing toward bringing marginally non-acceptable blocks up to an acceptable level of visual quality;

coding an enhancement-layer in response to structure analysis of the base-layer for acceptably synthesized blocks and using said alternate non-textural coding for unacceptably synthesized blocks; and

outputting at least one bit stream containing the coded base-layer and the enhancement-layer.

11. A method as recited in claim 10, wherein said performing of texture classification comprises characterizing each block in said downsampled video, with respect to its texture synthesis ability, in response to spatio-temporal characteristics of the downsampled video.

12. A method as recited in claim 10, wherein said alternate non-texture coding comprises conventional video coding utilizing AVC or MPEG-2 protocols.

13. A method as recited in claim 10, wherein said alternate non-textural coding comprises:

determining that a block is an intra-coded non-texture block, wherein the base-layer block is decoded, up-sampled, and a residual computed using the original input high-definition video signal; and

determining that a block is an inter-coded non-texture block, wherein the motion vector information (mv) from the base-layer is up-sampled, and

(i) for an mv in a non-synthesized region in the reference frame of the enhancement-layer, a refinement for mv is calculated, after which residual and refinement mv are coded, or

(ii) for an mv in a synthesized region of the reference frame of the enhancement-layer, the synthesized reference block is replaced with the current block, or

(iii) for an mv in a synthesized region of the reference frame of the enhancement-layer, a base-layer inter block is decoded, the downsampled video is up-sampled, and the residual from the enhancement-layer video is coded through at least one decoder.

14. A method as recited in claim 10, wherein said method is structurally aware in response to decoding of the base-layer to extract structural features.

15. A method as recited in claim 14, wherein said structural features comprise the extent of edges or structure as well as their position and direction.

16. A method as recited in claim 15, wherein structural features are preserved by obtaining structural information from the decoded base-layer which is not subject to texture synthesis.

17. A method as recited in claim 10, wherein said texture synthesis is performed in response to structural information about the presence, extent and positioning of edges and structures.

18. A method as recited in claim 10, wherein said texture synthesis utilizes low resolution texture information from the decoded base layer during the synthesis process of the high resolution layer.

19. An apparatus for efficient coding of a high-resolution video sequence, comprising:

a computer configured for processing high-definition video signals; and

programming executable on said computer for,

spatial downsampling of a high-resolution video sequence input to obtain downsampled video having a lower spatial resolution video signal,

coding of a base-layer of said downsampled video into a decoded base layer utilizing non-textural video coding,

performing texture analysis and classification,

determining structural information from said decoded base-layer,

performing texture synthesis on blocks, which are determined to be texture synthesis compatible, in response to said structural information from said decoded base layer,

coding an enhancement-layer in response to structural information from the coded base-layer for blocks which were acceptably synthesized, and using a non-textural coding framework for blocks which were not synthesized or for which acceptable synthesized results were not obtained, and

outputting at least one bit stream containing the coded base-layer and the enhancement-layer.

20. A computer-readable media containing a computer program executable on a computer configured for processing high-resolution video and causing the computer to generate scalable coding in response to steps, comprising:

spatial downsampling of a high-resolution video sequence received as input to obtain downsampled video having a lower spatial resolution video signal;

coding of a base-layer of said downsampled video into a decoded base layer utilizing non-textural video coding;

performing texture analysis and classification;

determining structural information from said decoded base-layer;

performing texture synthesis on blocks, which are determined to be texture synthesis compatible, in response to receipt said structural information obtained from said decoded base layer;

coding an enhancement-layer in response to structural information from the base-layer for blocks which were acceptably synthesized, and using a non-textural coding framework for blocks which were not synthesized or for which acceptable synthesized results were not obtained; and

outputting at least one bit stream containing the coded base-layer and the enhancement-layer.