Packetization of FGS/PFGS video bitstreams

• US 7,283,589 B2
• Filed: 03/10/2003
• Issued: 10/16/2007
• Est. Priority Date: 03/10/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Xiong, Zixiang, Li, Shipeng, Cai, Hua, Shen, Guobin, Zeng, Bing
• Primary Examiner(s): Vo; Tung
• Assistant Examiner(s): SENFI, BEHROOZ M
• Application Number:
  US10/385,014
• Publication Number:
  US 20040179617A1
• Time in Patent Office: 1,681 Days
• Field of Search: 348/387.1, 348/388.1, 348/554, 382/240, 382/232, 382/332, 375/240.26, 375/240.27, 375/240.16, 375/240.12, 375/240.01, 375/240.25
• US Class Current: 375/240.26
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/19 using optimisation based on Lagrange multipliers
• CPC Class Codes: H04N 19/34 Scalability techniques involving progressive bit-plane based encoding of the enhancement layer, e.g. fine granular scalability [FGS]
• CPC Class Codes: H04N 19/67 involving unequal error protection [UEP], i.e. providing protection according to the importance of the data
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 21/236 Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream multiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00

Abstract

A video encoding system performs packetization of FGS/PFGS encoded video bitstreams by selecting encoded bitstream segments for packetization based on an estimated total contribution to distortion reduction associated with each encoded bitstream segment. The selected bitstream segments are then packetized according to a packet-independence packetization strategy that minimizes inter-packet dependency.

Claims

1. A method comprising:

generating a plurality of encoded bitstream segments by applying a layered coding scheme to a video stream;

calculating a contribution to distortion reduction associated with each of the plurality of encoded video bitstream segments according to a formula;

Δ

D(f,l,i)=Δ

D₀(f,l,i)+Δ

D_dr(f,l,i)

wherein;

Δ

D₀(f,l,i) represents a distortion reduction of a bitstream segment associated with an f^th frame, an l^th bitplane, and an i^th macroblock; and

Δ

D_dr(f,l,i) represents a drifting suppression of the bitstream segment associated with the f^th frame l^th bitplane, and i^th macroblock;

selecting multiple bitstream segments from the plurality of encoded bitstream segments based on a comparison of the contribution to distortion reduction associated with individual ones of the encoded bitstream segments; and

packetizing the multiple bitstream segments into data packets such that a first data packet contains bitstream segments associated with multiple bit-planes of a first macro block, and such that no other one of the data packets contains bitstream segments associated with the first macro block.

2. The method as recited in claim 1 wherein the layered coding scheme comprises a fine granularity scalable coding scheme.

3. The method as recited in claim 1 wherein the layered coding scheme comprises a progressive fine granularity scalable coding scheme.

4. The method as recited in claim 3 wherein the progressive fine granularity scalable coding scheme is frame-based.

5. The method as recited in claim 3 wherein the progressive fine granularity scalable coding scheme is macroblock based.

6. The method as recited in claim 1 wherein the selecting is further based on an available bandwidth.

7. The method as recited in claim 1 wherein the selecting comprises:

determining a bitrate associated with individual ones of the plurality of encoded bitstream segments;

determining an available bandwidth;

and selecting the multiple bitstream segments such that for each bitstream segment that is selected, the associated bitrate is supported by the available bandwidth and the associated contribution to distortion reduction is greater than or equal to a contribution to distortion reduction associated with at least one of the plurality of bitstream segments that is not selected.

8. The method as recited in claim 1 wherein the first data packet also contains bitstream segments associated with multiple bit-planes of a second macro block.

31. The method as recited in claim 1 wherein the packetizing further comprises:

raster-scan ordering the bitstream segments in the first data packet according to the distortion reduction contribution of each bitstream segment associated with the first macro block.

9. A method comprising:

generating a plurality of encoded bitstream segments by applying a layered coding scheme to a video stream;

selecting multiple bitstream segments from the plurality of encoded bitstream segments;

representing multiple enhancement layers associated with a frame as a binary tree wherein each node of the tree represents a particular one of the multiple bitstream segments, and wherein each child node of the binary tree is aligned with an edge of a parent node; and

generating multiple data packets such that each data packet that contains a child node also contains a parent node associated with the child node.

27. The method as recited in claim 9 wherein the layered coding scheme comprises a fine granularity scalable coding scheme.

28. The method as recited in claim 9 wherein the selecting comprises:

selecting multiple bitstream segments from the plurality of encoded bitstream segments based on an available bandwidth.

29. The method as recited in claim 9 wherein the selecting comprises:

selecting multiple bitstream segments based on contributions to distortion reduction associated with individual ones of the encoded bitstream segments.

30. The method as recited in claim 9 wherein the selecting comprises:

determining a bitrate associated with individual ones of the plurality of encoded bitstream segments;

estimating a contribution to distortion reduction associated with individual ones of the plurality of encoded bitstream segments;

determining an available bandwidth; and

selecting the multiple bitstream segments such that for each bitstream segment that is selected, the associated bitrate is supported by the available bandwidth and the associated contribution to distortion reduction is greater than or equal to a contribution to distortion reduction associated with at least one of the plurality of bitstream segments that is not selected.

10. A method comprising:

identifying a plurality of encoded video bitstream segments to be packetized;

arranging the encoded video bitstream segments according to a binary tree that represents enhancement layers of a frame; and

packetizing the encoded video bitstream segments into data packets according to the binary tree such that each data packet comprises a parent node and a child node of the binary tree.

11. The method as recited in claim 10 further comprising:

pruning the binary tree to remove individual bitstream segments according to a distortion reduction contribution threshold.

12. A method comprising:

calculating a contribution to distortion reduction associated with each of a plurality of encoded video bitstream segments according to a formula;

Δ

D(f,l,i)=Δ

D₀(f,l,i)+Δ

D_dr(f,l,i)

wherein;

Δ

D₀(f,l,i) represents a distortion reduction of a bitstream segment associated with an f^th frame, an l^th bitplane, and an i^th macroblock; and

Δ

D_dr(f,l,i) represents a drifting suppression of the bitstream segment associated with the f^th frame, l^th bitplane, and i^th macroblock;

identifying for packetization, a subset of the plurality of encoded video bitstream segments based on a comparison of the contribution to distortion reduction; and

packetizing the subset of the plurality of encoded bitstream segments into data packets such that a first data packet contains bitstream segments associated with multiple bit-planes of a first macro block, and such that no other one of the data packets contains bitstream segments associated with the first macro block.

13. The method as recited in claim 12 wherein the packetizing further comprises:

raster-scan ordering the subset of the plurality of encoded bitstream segments in the first data packet according to the distortion reduction contribution of each bitstream segment associated with the first macro block.

14. A system comprising:

an encoder configured to generate encoded bitstream segments;

a bitstream selector configured to select a set of the encoded bitstream segments for packetization; and

a packet generator configured to packetize the set of the encoded bitstream segments according to a binary tree packetization strategy that minimizes dependency between packets.

15. The system as recited in claim 14, wherein the encoder is implemented as a fine granularity scalable encoder.

16. The system as recited in claim 14, wherein the encoder is implemented as a progressive fine granularity scalable encoder.

17. The system as recited in claim 14, wherein the encoder is implemented as a frame-based progressive fine granularity scalable encoder.

18. The system as recited in claim 14, wherein the encoder is implemented as a macroblock-based progressive fine granularity scalable encoder.

19. The system as recited in claim 14, wherein the encoder is further configured to gather data describing a position, a required bitrate, and a contribution to distortion reduction associated with an encoded bitstream segment.

20. A system comprising:

means for encoding streaming video data using a layered coding scheme, resulting in a plurality of encoded bitstream segments;

means for selecting a set of the encoded bitstream segments based on a contribution to distortion reduction associated with individual ones of the plurality of encoded bitstream segments, wherein the contribution to distortion reduction is calculated according to a formula;

Δ

D(f,l,i)=Δ

D₀(f,l,i)+Δ

D_dr(f,l,i)

wherein;

Δ

D₀(f,l,i) represents a distortion reduction of a bitstream segment associated with an f^th frame, an l^th bitplane, and an i^th macroblock; and

Δ

D_dr(f,l,i) represents a drifting suppression of the bitstream segment associated with the f^th frame, l^th bitplane, and i^th macroblock; and

means for generating data packets containing the set of encoded bitstream segments such that a first data packet contains encoded bitstream segments associated with multiple bit-planes of a first macro block, and such that no other one of the data packets contains bitstream segments associated with the first macro block, so that_the packetized bitstream segments are arranged to reduce dependency between individual ones of the data packets.

21. The system as recited in claim 20 wherein the generating means further comprises:

means for organizing the set of encoded bitstream segments within the data packets based on contributions to distortion reduction associated with individual encoded bitstream segments of the set of encoded bitstream segments.

22. The system as recited in claim 20 wherein the generating means further comprises:

means for organizing the set of encoded bitstream segments within the data packets in a decreasing order based on contributions to distortion reduction associated with individual encoded bitstream segments of the set of encoded bitstream segments.

23. The system as recited in claim 20 further comprising:

means for selecting the set of encoded bitstream segments based on an available bandwidth.

24. One or more computer-readable media comprising computer executable instructions that, when executed, direct a computing system to:

calculate the contribution to distortion reduction associated with each of a plurality of encoded video bitstream segments according to a formula;

Δ

D(f,l,i)=Δ

D₀(f,l,i)+Δ

D_dr(f,l,i)

wherein;

Δ

D₀(f,l,i) represents a distortion reduction of a bitstream segment associated with an f^th frame, an l^th bitplane, and an i^th macroblock; and

Δ

D_dr(f,l,i) represents a drifting suppression of the bitstream segment associated with the f^th frame, l^th bitplane, and i^th macroblock;

determine a subset of the plurality of encoded video bitstream segments based on a comparison of the contribution to distortion reduction; and

generate a data packet containing the selected subset of video bitstream segments.

25. The one or more computer-readable media as recited in claim 24 further comprising computer executable instructions that, when executed, direct a computing system to:

calculate the drifting suppression according to a formula;

$\begin{array}{c}\mathrm{\Delta <br><br>}<br><br><br><br>D_{d<br><br><br><br>r}<br><br>\left(f,l,i\right)=\mathrm{\Delta <br><br>}<br><br><br><br>D_0<br><br>\left(f,l,i\right)\times <br><br>\underset{n=1}{\overset{N}{\sum <br><br>}}<br><br><br><br>[\underset{k=1}{\overset{n}{\prod <br><br>}}<br><br><br><br>R_{\mathrm{HH}}<br><br>\left(f+k\right)+\\ <br><br>\underset{k=1}{\overset{n-1}{\prod <br><br>}}<br><br><br><br>R_{\mathrm{HH}}<br><br>\left(f+k\right)\times <br><br>R_{\mathrm{HL}}<br><br>\left(f+n\right)]\end{array}$

wherein;

Δ

D₀(f,l,i) represents a distortion reduction of a bitstream segment associated with an f^th frame, an l^th bitplane, and an i^th macroblock;

R_HH(f) represents a percentage of high-prediction-high-reference (HPHR) modes in an f^th frame;

R_HL(f) represents a percentage of high-prediction-low-reference (HPLR) modes in an f^th frame; and

N is a number of frames used for drifting estimation.

26. The one or more computer-readable media as recited in claim 24, wherein each of the plurality of encoded video bitstream segments is encoded according to a layered coding scheme.