Streaming Methods and Systems

• US 20090122878A1
• Filed: 01/19/2009
• Published: 05/14/2009
• Est. Priority Date: 04/11/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Hong-Jiang, Qi, Wei, Liu, Tianming
• Granted Patent: US 8,144,769 B2
• Time in Patent Office: Days
• US Class Current: 375/240.260
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/34 Scalability techniques involving progressive bit-plane based encoding of the enhancement layer, e.g. fine granular scalability [FGS]
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/87 involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/2401 Monitoring of the client buffer
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/6373 for rate control , e.g. request to the server to modify its transmission rate flow control in packet networks H04L47/10

Abstract

Various embodiments provide methods and systems for streaming data that can facilitate streaming during bandwidth fluctuations in a manner that can enhance the user experience. In one aspect, a forward-shifting technique is utilized to buffer data that is to be streamed, e.g. an enhancement layer in a FGS stream. Various techniques can drop layers actively when bandwidth is constant. The saved bandwidth can then be used to pre-stream enhancement layer portions. In another aspect, a content-aware decision can be made as to how to drop enhancement layers when bandwidth decreases. During periods of decreasing bandwidth, if a video segment does not contain important content, the enhancement layers will be dropped to keep the forward-shifting of the enhancement layer unchanged. If the enhancement layer does contain important content, it will be transmitted later when bandwidth increases.

Claims

1. A method comprising:

providing a streaming server configured to stream encoded data, the encoded data comprising a plurality of layers of data; and

determining whether portions of a secondary (enhancement) layer are important;

deciding to drop enhancement layer portions based, at least in part, on whether the portions are important,

dropping at least portions of the enhancement layer of the streamed encoded data sufficient to forward-shift, relative to a primary layer, the subsequent portions of the enhancement layer by an amount.

2. The method of claim 1 further comprising:

determining a client side buffer size; and

deciding to drop enhancement layer portions based, at least in part, on the client side buffer size.

3. The method of claim 1, wherein dropping portions of the enhancement layer comprises dropping portions that are determined to be less important than portions that are not dropped.

4. The method of claim 1 further comprising:

in an event the enhancement layer contains content that is determined to be important and bandwidth is insufficient, delaying transmission of portions of the enhancement layer that contain the important content.

5. The method of claim 1, wherein the dropping comprises dropping at least one entire enhancement layer.

6. One or more computer-readable media having computer-readable instructions thereon which, when executed by one or more processors, cause the one or more processors to implement the method of claim 1.

7. A method comprising:

dropping portions of an enhancement layer to forward-shift following portions of the enhancement layer by an amount;

determining that a decrease in available bandwidth has occurred;

responsive to said determining;

determining whether the enhancement layer contains important content;

if it is determined that the enhancement layer does not contain important content, dropping additional portions of the enhancement layer to preserve a forward-shifted enhancement layer; and

is if it is determined that the enhancement layer does contain important content, delaying transmission of portions of the enhancement layer so that the delayed portions can be transmitted when the available bandwidth increases.

8. The method of claim 7, wherein said act of first dropping occurs prior to streaming any enhancement layer portions.

9. The method of claim 7 further comprising:

determining that the available bandwidth has increased;

responsive to determining that the available bandwidth has increased, transmitting the delayed portions of the enhancement layer; and

after said transmitting of the delayed portions, dropping additional portions of the enhancement layer sufficient to forward shift following portions of the enhancement layer by an amount.

10. The method of claim 7, wherein said acts of dropping comprise dropping one or more enhancement layers.

11. One or more computer-readable media having computer-readable instructions thereon which, when executed by one or more processors, cause the one or more processors to implement the method of claim 7.

12. A streaming server comprising:

one or more processors;

memory; and

software code embodiment in the memory which, when executed by the one or more processors, cause the one or more processors to drop at least portions of an enhancement layer sufficient to forward-shift following portions of the enhancement layer by an amount.

13. The streaming server of claim 12, wherein the software code causes the one or more processors to:

determine whether portions of the enhancement layer are important; and

make a decision to drop enhancement layer portions based, at least in part, on whether the portions are important.

14. The streaming server of claim 12, wherein the software code causes the one or more processors to:

determine a client side buffer size; and

make a decision to drop enhancement layer portions based, at least in part, on the client side buffer size.

15. The streaming server of claim 12, wherein the software code causes the one or more processors to drop portions of the enhancement layer that are less important than portions that are not dropped.

16. The streaming server of claim 12, wherein the software code causes the one or more processors to:

determine whether a change in available bandwidth has occurred;

responsive to determining that the available bandwidth has decreased;

determine whether the enhancement layer contains is important content, and;

in the event that the enhancement layer does not contain important content, drop additional portions of the enhancement layer in an attempt to preserve a forward-shifted enhancement layer;

in the event that the enhancement layer does contain important content, delay transmission of portions of the enhancement layer so that the delayed portions can be transmitted when the available bandwidth increases.

responsive to determining that the available bandwidth has increased;

transmit the delayed portions of the enhancement layer; and

drop additional portions of the enhancement layer sufficient to forward shift subsequent portions of the enhancement layer by an amount.

17. The streaming server of claim 12, wherein the software code causes the one or more processors to drop one or more entire enhancement layers.