Spatial scalability for fine granular video encoding

• US 6,836,512 B2
• Filed: 10/11/2001
• Issued: 12/28/2004
• Est. Priority Date: 10/11/2000
• Status: Expired due to Fees

• Current Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: Balakrishnan, Mahesh, Van Der Schaar, Mihaela
• Primary Examiner(s): Le, Vu
• Application Number:
  US09/975,626
• Publication Number:
  US 20020071486A1
• Time in Patent Office: 1,174 Days
• Field of Search: 375/240.02, 375/240.08, 375/240.11, 375/240.18, 375/240.19, 375/240.2, 375/240.21, 382/236-237, 382/238, 382/240, 382/248, 348/398.1, 348/397.1, 348/399.1, 348/408.1, 348/424.1
• US Class Current: 375/240.11
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/34 Scalability techniques involving progressive bit-plane based encoding of the enhancement layer, e.g. fine granular scalability [FGS]
• CPC Class Codes: H04N 19/36 Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/234354 by altering signal-to-noise ratio parameters, e.g. requantization
• CPC Class Codes: H04N 21/234363 by altering the spatial resolution, e.g. for clients with a lower screen resolution
• CPC Class Codes: H04N 21/234381 by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities

Abstract

A method and system for high resolution formatting of video images and dynamically adjusting the transmission resolution of the high-resolution images is presented. The method first downscales the high-resolution images and encodes the downscaled images into base layer frames. Quality enhancement layer data is generated from the downscaled video images and the encoded data contained in corresponding base layer frames. The quality enhancement layer data is encoded into quality enhancement layer frames. The data contained in the base layer frames and corresponding quality layer frames are then upscaled and spatial scalability data is determined from the upscaled data and the original image. The spatial scalability data is then encoded into spatial scalability data. During transmission of the encoded video image, each available encoded frame is transmitted using different amounts or portions of the enhancement layers so as to occupy the available bandwidth.

Claims

1. A method for coding video data contained in image frames comprising the steps of:

downscaling said video data image frames;

encoding said downscaled video data to produce base layer frames;

generating quality enhanced residual images from said downscaled video data and said base layer frames;

encoding said quality enhanced residual images using a fine granular coding technique to produce quality enhancement layer frames;

upscaling said base layer and said quality enhanced residual frames;

generating a first set of residual images from said upscaled encoded data contained base layer, and corresponding quality enhancement layer frames and said video data;

encoding said first set of residual images using a fine granular coding technique to produce spatial enhancement layer frames.

2. The method as recited in claim 1 further comprising the step of;

generating a second set of residual images from said first set of residual images; and

coding said second set of residual images using a fine granular coding technique to produce temporal enhancement layer frames.

3. The method as recited in claim 1, further comprising the step of:

transmitting said base layer frames, and portions of said spatial enhancement frames over a variable bandwidth network.

4. The method as recited in claim 3 further comprising the steps of:

determining an available bandwidth of said network;

selecting the portions of said spatial enhancement layer frames to be transmitted based on said available bandwidth.

5. The method as recited in claim 3, further comprising the step of:

transmitting portions of said temporal enhancement layer frames over a variable bandwidth network.

6. The method as recited in claim 5 further comprising the steps of:

determining an available bandwidth of said network;

selecting the portions of said temporal enhancement layer frames to be transmitted based on said available bandwidth.

7. The method as recited in claim 1 further comprises the step of:

combing said quality enhancement frames and said spatial enhancement layer frames into an enhancement frame layer.

8. The method as recited in claim 7 further comprises the step of:

transmitting said base layer frames and portions of said combined quality enhancement layer frames and said spatial enhancement layer frames over a variable bandwidth network.

9. The method as recited in claim 8 further includes determining an available bandwidth of the network;

and

selecting portions of said quality enhancement layer and portions of said spatial layer frames to be transmitted based on said available bandwidth.

10. The method as recited in claim 8 further comprises the step of:

combing said enhancement layer frame and said temporal enhancement layer frames into a second enhancement frame layer.

11. The method as recited in claim 10 further comprises the step of:

transmitting said base layer frames and portions of said combined quality enhancement layer frames, spatial enhancement layer frames and temporal enhancement layer frames over a variable bandwidth network.

12. The method as recited in claim 11 further includes determining an available bandwidth of the network;

and

selecting portions of said quality enhancement layer and portions of said spatial layer frames to be transmitted based on said available bandwidth.

13. The method as recited in claim 1 wherein said spatial enhancement layer frames are motion compensated.

14. The method as recited in claim 2 wherein said temporal enhancement layer frames are motion compensated.

15. A method for coding video data comprising the steps of:

downscaling said video data;

coding said downscaled video data to produce base layer frames;

generating residual images from said downscaled video data and said base layer frames;

coding said residual images using a fine granular coding technique to produce quality enhancement layer frames;

upscaling said base layer and said quality enhancement layer frames

generating a second set of residual images from upscaled base layer and said quality enhancement frames and said video data;

coding said second residual images using a fine granular coding technique to produce spatial enhancement layer frames.

16. The method as recited in claim 15 further comprising the steps of:

generating a third set of residual images from corresponding spatial layers; and

coding said third set of residual images using a fine granular coding technique to produce temporal enhancement layer frames.

17. The method as recited in claim 15 wherein said spatial enhancement layer frames are motion compensated.

18. The method as recited in claim 16 wherein said temporal enhancement layer frames are motion compensated.

19. A memory medium including code for encoding video data, said code comprising:

code to downscale said video data;

code to encode said downscaled video data to produce base layer frames;

code to generate residual images from said downscaled video data and said base layer to produce quality enhancement frames;

code to upscale said base layer frames and said quality enhancement layer frames;

code to generated a second residual images from upscaled frames and said video data; and

code to encode said second residual images using a fine granular coding technique to produce spatial enhancement layer frames.

20. A memory medium as recited in claim 19 further comprising:

code to generate a third residual images from said spatial enhancement frames;

code to encode said third residual images using a fine granular coding technique to produce temporal enhancement layer frames.

21. A memory medium as recited in claim 19 wherein said spatial enhancement frames are motion compensated.

22. A memory medium as recited in claim 20 wherein said temporal enhancement layer frames are motion compensated.

23. An apparatus for coding video data contained in image frames comprising:

a downscaler to downscale said video data image frames;

an encoder operable to;

encode said downscaled video data to produce base layer frames;

generate quality enhanced residual images from said downscaled video data and said base layer frames;

encode said quality enhanced residual images using a fine granular coding technique to produce quality enhancement frames

an upscaler to upscale said base layer and said quality enhanced residual images;

a second encoder operable to;

generate a first set of residual images from said upscaled base layer, said upscaled residual images and said video data;

encode said first set of residual images using a fine granular coding technique to produce spatial enhancement layer frames.

24. The apparatus as recited in claim 23 wherein said second encoder is further operable to:

generate a second set of residual images from said first set of residual images; and

encode said second set of residual images using a fine granular coding technique to produce temporal enhancement layer frames.

25. The apparatus as recited in claim 23, further comprising

a transmitting device to transmit said base layer frames, and portions of said spatial enhancement frames over a variable bandwidth network.

26. The apparatus as recited in claim 25 further operable to:

determine an available bandwidth of said network;

select portions of said spatial enhancement frames to be transmitted based on said available bandwidth.

27. The apparatus as recited in claim 25, further operable to:

transmit portions of said temporal enhancement layer frames over a variable bandwidth network.

28. The apparatus as recited in claim 27 further operable to:

determine an available bandwidth of said network;

select portions of said temporal enhancement layer frames to be transmitted based on said available bandwidth.

29. The apparatus as recited in claim 23 further operable to:

combine said quality enhancement layer frames and said spatial enhancement layer frames into a combined enhancement layer frame.

30. The apparatus as recited in claim 29 further operable to:

transmit said base layer frames and portions of said combined enhancement layer frames over a variable bandwidth network.

31. The apparatus as recited in claim 27 further operable to:

determine an available bandwidth of the network; and

select portions of said quality enhancement layer and portions of said spatial layer frames to be transmitted based on said available bandwidth.

32. The apparatus as recited in claim 30 further operable to:

combine said enhancement frame layer and said temporal enhancement layer frames into a second enhancement layer frame.

33. The apparatus as recited in claim 32 further operable to:

transmit said base layer frames and portions of said combined quality enhancement frames, spatial enhancement frames and temporal enhancement frames over a variable bandwidth network.

34. The apparatus as recited in claim 33 further operable to

determine an available bandwidth of the network;

and

select portions of said quality enhancement layer and portions of said spatial layer frames to be transmitted based on said available bandwidth.

35. The apparatus as recited in claim 27 wherein said spatial enhancement layer frames are motion compensated.

36. The apparatus as recited in claim 28 wherein said temporal enhancement layer frames are motion compensated.

37. A method for coding video data comprising the steps of:

downscaling said video data;

coding said downscaled video data to produce base layer frames;

generating residual images from among said base layer frames;

coding said residual images using a fine granular coding technique to produce first temporal enhancement layer frames

generating residual images from said downscaled video data and said base layer frames;

coding said residual images using a fine granular coding technique to produce quality enhancement frames;

upscaling said base layer and said quality enhancement layer frames

generating a second set of residual images from said upscaled base layer and said quality enhancement layer frames and said video data;

coding said second residual images using a fine granular coding technique to produce spatial enhancement layer frames.

38. The method as recited in claim 37 further comprising the steps of:

generating a third set of residual images from among corresponding spatial layers; and

coding said third set of residual images using a fine granular coding technique to produce second temporal enhancement layer frames.

39. The method as recited in claim 37 further comprises the step of:

transmitting said base layer frames and portions of said combined quality enhancement layer frames, said spatial enhancement frames and said first temporal enhancement layer frames over a variable bandwidth network.

40. The method as recited in claim 38 further comprises the step of:

transmitting said base layer frames and portions of said combined quality enhancement frames, said spatial enhancement frames, said first temporal and said second temporal enhancement layer frames over a variable bandwidth network.

41. The method as recited in claim 37 wherein said spatial enhancement layer frames are motion compensated.

42. The method as recited in claim 38 wherein said temporal enhancement layer frames are motion compensated.

43. A system for coding video data comprising:

means for downscaling said video data;

means for coding said downscaled video data to produce base layer frames;

means for generating residual images from said downscaled video data and said base layer frames;

means for coding said residual images to produce quality enhancement frames;

means for upscaling said base layer and said quality enhancement frames

means for generating a second set of residual images from upscaled base layer and said quality enhancement frames and said video data;

means for coding said second residual images to produce spatial enhancement layer frames.

44. The system as recited in claim 43 further comprising:

means for generating a third set of residual images from corresponding spatial layers; and

means for coding said third set of residual images to produce temporal enhancement layer frames.

45. The system as recited in claim 43 wherein said means for coding includes FGS encoding.

46. The system as recited in claim 44 wherein said means for coding includes FGS encoding.