Architecture and method for fine granularity scalable video coding

• US 20040042549A1
• Filed: 08/27/2002
• Published: 03/04/2004
• Est. Priority Date: 08/27/2002
• Status: Active Grant

• Current Assignee: National Chiao Tung University (Government of The Republic of China)
• Original Assignee: National Chiao Tung University (Government of The Republic of China)
• Inventors: Huang, Hsiang-Chun, Chiang, Tihao, Hang, Hsueh-Ming, Wang, Chung-Neng
• Granted Patent: US 7,072,394 B2
• Time in Patent Office: Days
• US Class Current: 375/240.11
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/34 Scalability techniques involving progressive bit-plane based encoding of the enhancement layer, e.g. fine granular scalability [FGS]

Abstract

A robust fine granularity scalability video encoding includes a base layer encoder and an enhancement layer encoder in which motion compensated difference images are generated by comparing an original image to predicted images at base layer and enhancement layer with motion compensation. Based on leaky and partial predictions, a high quality reference image is constructed at the enhancement layer to improve temporal prediction. In the construction of the high quality reference image, one parameter β controls the number of bitplanes of the enhancement layer difference coefficients used and another parameter α controls the amount of predictive leak. A spatial scalability module allows the processed pictures at the base layer and the enhancement layer to have identical or different spatial resolutions.

Claims

1. An apparatus for robust fine granularity scalability video encoding, comprising:

a base layer encoder receiving an original video image and a base layer reference image for generating a base layer bit stream; and

an enhancement layer encoder receiving said original video image, a motion estimation vector, a base layer reconstructed image, and coded base layer discrete cosine transform coefficients for generating an enhancement layer bit stream;

wherein said base layer reference image is generated from said enhancement layer encoder or said base layer encoder, and said base layer reconstrcuted image is generated from said base layer encoder.

2. The apparatus for robust fine granularity scalability video encoding as claimed in claim 1, said base layer encoder further comprising:

a motion estimator receiving said original video image and said base layer reference image for estimating a motion vector;

a motion compensator receiving said base layer reference image and said motion vector for generating a base layer predicted image;

a subtraction unit subtracting said base layer predicted image from said original video image and generating a motion compensated base layer frame difference image;

a discrete cosine transform unit transforming said motion compensated base layer frame difference image into motion compensated base layer difference coefficients;

a quantizer quantizing said motion compensated base layer difference coefficients into quantized base layer difference coefficients; and

a variable length coding unit receiving said motion vector and said quantized base layer difference coefficients for generating said base layer bit stream.

3. The apparatus for robust fine granularity scalability video encoding as claimed in claim 2, said base layer encoder further comprising:

a dequantizer receiving said quantized base layer difference coefficients and generating said coded base layer discrete cosine transform coefficients;

an inverse discrete cosine transform unit receiving said coded base layer discrete cosine transform coefficients and generating a decoded base layer difference image;

a summation unit adding said decoded base layer difference image to said base layer predicted image and generating an unclipped base layer reconstructed image; and

a clipping unit receiving said unclipped base layer reconstructed image and generating said base layer reconstructed image; and

a base layer frame buffer for buffering said base layer reconstructed image.

4. The apparatus for robust fine granularity scalability video encoding as claimed in claim 3, said enhancement layer encoder further comprising:

a motion compensator receiving said motion vector and a buffered enhancement layer reconstructed image for generating an enhancement layer predicted image;

a subtraction unit subtracting said enhancement layer predicted image from said original video image and generating a motion compensated enhancement layer difference image;

a discrete cosine transform unit transforming said motion compensated enhancement layer difference image into motion compensated enhancement layer difference coefficients;

a subtraction unit subtracting said coded base layer discrete cosine transform coefficients from said motion compensated enhancement layer difference coefficients and generating first residue coefficients;

a subtraction unit subtracting said coded base layer discrete cosine transform coefficients from said motion compensated base layer difference coefficients and generating second residue coefficients;

a bitplane coding unit receiving said first or second residue coefficients and generating enhancement layer difference coefficients; and

a variable length coding unit receiving said enhancement layer difference coefficients and generating an enhancement layer bit stream.

5. The apparatus for robust fine granularity scalability video encoding as claimed in claim 4, said enhancement layer encoder further comprising:

a summation unit adding said coded base layer discrete cosine transform coefficients to first β

bitplanes of said enhancement layer difference coefficients for generating enhancement layer partial difference coefficients;

an inverse discrete cosine transform unit transforming-said enhancement layer partial difference coefficients into an enhancement layer partial difference image;

a summation unit adding said enhancement layer partial difference image to said enhancement layer predicted image and generating an unclipped enhancement layer partially reconstructed image;

a clipping unit receiving said unclipped enhancement layer partially reconstructed image and generating an enhancement layer partially reconstructed image;

a subtraction unit subtracting said base layer reconstructed image from said enhancement layer partially reconstructed image and generating an enhancement layer reconstructed difference image;

a scaler scaling said enhancement layer reconstructed difference image by a factor α and

generating a scaled enhancement layer reconstructed difference image;

a summation unit adding said scaled enhancement layer reconstructed difference image to said base layer reconstructed image and generating an enhancement layer reconstructed image; and

an enhancement layer frame buffer for buffering said enhancement layer reconstructed image.

6. The apparatus for robust fine granularity scalability video encoding as claimed in claim 5, wherein said base layer reference image is selected from an image buffered in said base layer frame buffer or said enhancement layer frame buffer.

7. The apparatus for robust fine granularity scalability video encoding as claimed in claim 5, wherein β

is an integer number between 0 and a maximum number of bit planes of said enhancement layer difference coefficients.

8. The apparatus for robust fine granularity scalability video encoding as claimed in claim 5, wherein α

is greater than or equal to 0 and smaller than or equal to 1.

9. The apparatus for robust fine granularity scalability video encoding as claimed in claim 5, further including a high quality base layer prediction module comprising:

a summation unit adding said coded base layer discrete cosine transform coefficients to first γ

bitplanes of said enhancement layer difference coefficients for generating predicted high quality enhancement layer partial difference coefficients;

an inverse discrete cosine transform unit transforming said predicted high quality enhancement layer partial difference coefficients into a predicted high quality enhancement layer partial difference image;

a summation unit adding said predicted high quality enhancement layer partial difference image to said enhancement layer predicted image and generating an unclipped high quality enhancement layer partially reconstructed image;

a clipping unit receiving said unclipped high quality enhancement layer partially reconstructed image and generating a high quality enhancement layer partially reconstructed image;

a subtraction unit subtracting said base layer reconstructed image from said high quality enhancement layer partially reconstructed image and generating a high quality enhancement layer reconstructed difference image;

a scaler scaling said high quality enhancement layer reconstructed difference image by a factor ζ and

generating a scaled high quality enhancement layer reconstructed difference image; and

a summation unit adding said scaled high quality enhancement layer reconstructed difference image to said base layer reconstructed image and generating a high quality base layer reference image;

wherein said base layer frame buffer further has a selection switch for selecting either said base layer reconstructed image or said high quality base layer reference image.

10. The apparatus for robust fine granularity scalability video encoding as claimed in claim 9, wherein γ

is an integer number between 0 and said maximum number of bit planes of said enhancement layer difference coefficients.

11. The apparatus for robust fine granularity scalability video encoding as claimed in claim 9, wherein ζ

is greater than or equal to 0 and smaller than or equal to 1.

12. An apparatus for robust fine granularity scalability video encoding, comprising:

a base layer encoder receiving an original video image and a base layer reference image for generating a base layer bit stream;

an enhancement layer encoder receiving a video image, a motion estimation vector, a base layer reconstructed image, and coded base layer discrete cosine transform coefficients for generating an enhancement layer bit stream; and

an encoder spatial scalability module having a plurality of decimators, a plurality of interpolators and a plurality of switches;

wherein said base layer reference image is generated from said enhancement layer encoder or said base layer encoder, said base layer reconstructed image is generated from said base layer encoder, and said encoder spatial scalability module controls if said base layer reference image is decimated or not when said base layer reference image is generated from said enhancement layer encoder and controls if said base layer reconstructed image is interpolated or not.

13. The apparatus for robust fine granularity scalability video encoding as claimed in claim 12, said base layer encoder further comprising:

a motion estimator receiving said original video image and said base layer reference image for estimating a motion vector;

a motion compensator receiving said base layer reference image and said motion vector for generating a base layer predicted image;

a subtraction unit subtracting said base layer predicted image from said original video image and generating a motion compensated base layer frame difference image;

a discrete cosine transform unit transforming said motion compensated base layer frame difference image into motion compensated base layer difference coefficients;

a quantizer quantizing said motion compensated base layer difference coefficients into quantized base layer difference coefficients; and

a variable length coding unit receiving said motion vector and said quantized base layer difference coefficients for generating said base layer bit stream.

14. The apparatus for robust fine granularity scalability video encoding as claimed in claim 13, said base layer encoder further comprising:

a dequantizer receiving said quantized base layer difference coefficients and generating said coded base layer discrete cosine transform coefficients;

an inverse discrete cosine transform unit receiving said coded base layer discrete cosine transform coefficients and generating a decoded base layer difference image;

a summation unit adding said decoded base layer difference image to said base layer predicted image and generating an unclipped base layer reconstructed image; and

a clipping unit receiving said unclipped base layer reconstructed image and generating said base layer reconstructed image; and

a base layer frame buffer for buffering said base layer reconstructed image.

15. The apparatus for robust fine granularity scalability video encoding as claimed in claim 14, said enhancement layer encoder further comprising:

a motion compensator receiving a motion vector and a buffered enhancement layer reconstructed image for generating an enhancement layer predicted image;

a subtraction unit subtracting said enhancement layer predicted image from a video image and generating a motion compensated enhancement layer difference image;

a discrete cosine transform unit transforming said motion compensated enhancement layer difference image into motion compensated enhancement layer difference coefficients;

a subtraction unit subtracting said coded base layer discrete cosine transform coefficients or interpolated base layer discrete cosine transform coefficients from said motion compensated enhancement layer difference coefficients and generating first residue coefficients;

a subtraction unit subtracting said coded base layer discrete cosine transform coefficients or said interpolated base layer discrete cosine transform coefficients from said motion compensated base layer difference coefficients and generating second residue coefficients;

a bitplane coding unit receiving said first or second residue coefficients and generating enhancement layer difference coefficients; and

a variable length coding unit receiving said enhancement layer difference coefficients and generating an enhancement layer bit stream, said variable length coding unit having an input for receiving an enhancement layer motion vector.

16. The apparatus for robust fine granularity scalability video encoding as claimed in claim 15, said enhancement layer encoder further comprising:

a summation unit adding said coded base layer discrete cosine transform coefficients or said interpolated base layer discrete cosine transform coefficients to first β

bitplanes of said enhancement layer difference coefficients for generating enhancement layer partial difference coefficients;

an inverse discrete cosine transform unit transforming said enhancement layer partial difference coefficients into an enhancement layer partial difference image;

a summation unit adding said enhancement layer partial difference image to said enhancement layer predicted image and generating an unclipped enhancement layer partially reconstructed image;

a clipping unit receiving said unclipped enhancement layer partially reconstructed image and generating an enhancement layer partially reconstructed image;

a subtraction unit subtracting said base layer reconstructed image or an interpolated base layer reconstructed image from said enhancement layer partially reconstructed image and generating an enhancement layer reconstructed difference image;

a scaler scaling said enhancement layer reconstructed difference image by a factor α and

generating a scaled enhancement layer reconstructed difference image;

a summation unit adding said base layer reconstructed image or said interpolated base layer reconstructed image to said scaled enhancement layer reconstructed difference image and generating an enhancement layer reconstructed image; and

an enhancement layer frame buffer for buffering said enhancement layer reconstructed image.

17. The apparatus for robust fine granularity scalability video encoding as claimed in claim 16, wherein said encoder spatial scalability module further comprises:

a decimator for decimating an image buffered in said enhancement layer frame buffer into a base layer reference image;

an interpolator for interpolating said base layer reconstructed image into an interpolated base layer reconstructed image;

an interpolator for interpolating said decoded base layer difference image into an interpolated base layer difference image;

a discrete cosine transform unit transforming said interpolated base layer difference image into said interpolated base layer discrete cosine transform coefficients; and

a motion estimator estimating said enhancement layer motion vector from an image buffered in said enhancement layer frame buffer.

18. The apparatus for robust fine granularity scalability video encoding as claimed in claim 16, wherein said base layer reference image is generated from an image buffered in said base layer frame buffer, an image buffered in said enhancement layer frame buffer, or a decimated image of an image buffered in said enhancement layer frame buffer.

19. The apparatus for robust fine granularity scalability video encoding as claimed in claim 16, wherein β

is an integer number between 0 and a maximum number of bit planes of said enhancement layer difference coefficients.

20. The apparatus for robust fine granularity scalability video encoding as claimed in claim 16, wherein α

is greater than or equal to 0 and smaller than or equal to 1.

21. The apparatus for robust fine granularity scalability video encoding as claimed in claim 16, further including a high quality base layer prediction module comprising:

a summation unit adding said coded base layer discrete cosine transform coefficients to first γ

bitplanes of said enhancement layer difference coefficients for generating predicted high quality enhancement layer partial difference coefficients;

an inverse discrete cosine transform unit transforming said predicted high quality enhancement layer partial difference coefficients into a predicted high quality enhancement layer partial difference image;

a summation unit adding said predicted high quality enhancement layer partial difference image to said enhancement layer predicted image and generating an unclipped high quality enhancement layer partially reconstructed image;

a clipping unit receiving said unclipped high quality enhancement layer partially reconstructed image and generating a high quality enhancement layer partially reconstructed image;

a subtraction unit subtracting said base layer reconstructed image from said high quality enhancement layer partially reconstructed image and generating a high quality enhancement layer reconstructed difference image;

a scaler scaling said high quality enhancement layer reconstructed difference image by a factor ζ and

generating a scaled high quality enhancement layer reconstructed difference image; and

a summation unit adding said scaled high quality enhancement layer reconstructed difference image to said base layer reconstructed image and generating a high quality base layer reference image;

wherein said base layer frame buffer further has a selection switch for selecting either said base layer reconstructed image, said high quality base layer reference image or a decimated high quality base layer reference image.

22. The apparatus for robust fine granularity scalability video encoding as claimed in claim 21, wherein γ

is an integer number between 0 and said maximum number of bit planes of said enhancement layer difference coefficients.

23. The apparatus for robust fine granularity scalability video encoding as claimed in claim 21, wherein ζ

is greater than or equal to 0 and smaller than or equal to 1.

24. The apparatus for robust fine granularity scalability video encoding as claimed in claim 21, wherein said encoder spatial scalability module further comprises a decimator for decimating said high quality base layer reference image into a decimated high quality base layer reference image.

25. An apparatus for robust fine granularity scalability video decoding, comprising:

a base layer decoder receiving a base layer bit stream and a base layer reference image for generating a base layer video image; and

an enhancement layer decoder receiving an enhancement layer bit stream, a motion estimation vector, and coded base layer discrete cosine transform coefficients for generating a display video image;

wherein said base layer reference image is generated from said enhancement layer decoder or said base layer decoder, and said coded base layer discrete cosine transform coefficients are generated from said base layer decoder.

26. The apparatus for robust fine granularity scalability video decoding as claimed in claim 25, said base layer decoder further comprising:

a variable length decoder receiving said base layer bit stream and generating a motion vector and quantized base layer difference coefficients;

a dequantizer receiving said quantized base layer difference coefficients and generating base layer difference coefficients;

an inverse discrete cosine transform unit receiving said base layer difference coefficients and generating a base layer difference image;

a motion compensator receiving said base layer reference image and said motion vector for generating a base layer predicted image;

a summation unit adding said base layer predicted image to said base layer difference image and generating an unclipped base layer video image; and

a clipping unit receiving said unclipped base layer video image and generating said base layer video image.

27. The apparatus for robust fine granularity scalability video decoding as claimed in claim 25, said enhancement layer decoder further comprising:

a variable length decoder receiving said enhancement layer bit stream and generating enhancement layer difference coefficients;

a bitplane decoder decoding said enhancement layer difference coefficients;

a summation unit adding decoded enhancement layer difference coefficients from said bitplane decoder to said base layer difference coefficients and generating enhancement layer discrete cosine transform coefficients;

an inverse discrete cosine transform unit receiving said enhancement layer discrete cosine transform coefficients and generating an enhancement layer difference image;

a summation unit adding said enhancement layer difference image to an enhancement layer predicted image and generating an unclipped display video image; and

a clipping unit clipping said unclipped display video image and generating a display video image.

28. The apparatus for robust fine granularity scalability video decoding as claimed in claim 27, said enhancement layer decoder further comprising:

a summation unit adding first β

bitplanes of said decoded enhancement layer difference coefficients from said bitplane decoder to said base layer difference coefficients and generating enhancement layer partial discrete cosine transform coefficients;

an inverse discrete cosine transform unit receiving said enhancement layer partial discrete cosine transform coefficients and generating an enhancement layer partial difference image;

a summation unit adding said enhancement layer partial difference image to said enhancement layer predicted image and generating an unclipped partial display video image;

a clipping unit clipping said unclipped partial display video image and generating a partial display video image;

a subtraction unit subtracting said base layer video image from said partial display video image and generating an enhancement layer reconstructed difference image;

a scaler scaling said enhancement layer reconstructed difference image by a factor α and

generating a scaled enhancement layer reconstructed difference image;

a summation unit adding said scaled enhancement layer reconstructed difference image to said base layer video image and generating an enhancement layer reconstructed image;

an enhancement layer frame buffer for buffering said enhancement layer reconstructed image;

a motion compensator receiving said motion vector and an enhancement layer reconstructed image buffered in said enhancement layer frame buffer and generating said enhancement layer predicted image.

29. The apparatus for robust fine granularity scalability video decoding as claimed in claim 28, wherein said base layer reference image is selected from an image buffered in said base layer frame buffer or said enhancement layer frame buffer.

30. The apparatus for robust fine granularity scalability video decoding as claimed in claim 28, wherein β

is an integer number between 0 and a maximum number of bit planes of said enhancement layer difference coefficients.

31. The apparatus for robust fine granularity scalability video decoding as claimed in claim 28, wherein α

is greater than or equal to 0 and smaller than or equal to 1.

32. The apparatus for robust fine granularity scalability video decoding as claimed in claim 28, further including a high quality base layer prediction module comprising:

a summation unit adding said base layer difference coefficients to first γ

bitplanes of said enhancement layer difference coefficients for generating predicted high quality enhancement layer partial difference coefficients;

an inverse discrete cosine transform unit transforming said predicted high quality enhancement layer partial difference coefficients into a predicted high quality enhancement layer partial difference image;

a summation unit adding said predicted high quality enhancement layer partial difference image to said enhancement layer predicted image and generating an unclipped high quality enhancement layer partially reconstructed image;

a clipping unit receiving said unclipped high quality enhancement layer partially reconstructed image and generating a high quality enhancement layer partially reconstructed image;

a subtraction unit subtracting said base layer video image from said high quality enhancement layer partially reconstructed image and generating a high quality enhancement layer reconstructed difference image;

a scaler scaling said high quality enhancement layer reconstructed difference image by a factor ζ and

generating a scaled high quality enhancement layer reconstructed difference image; and

a summation unit adding said scaled high quality enhancement layer reconstructed difference image to said base layer video image and generating a high quality base layer reference image;

wherein said base layer frame buffer further has a selection switch for selecting either said base layer video image or said high quality base layer reference image.

33. The apparatus for robust fine granularity scalability video decoding as claimed in claim 32, wherein γ

is an integer number between 0 and said maximum number of bit planes of said enhancement layer difference coefficients.

34. The apparatus for robust fine granularity scalability video decoding as claimed in claim 32, wherein ζ

is greater than or equal to 0 and smaller than or equal to 1.